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In This Issue 


This issue of the Survey Methodology journal contains a special memorial section in honour of 
Stanley L. Warner, which includes an introduction by C.-E. Sarndal, a bibliography of Warner’s 
principal publications and papers, organized by topic, and three papers dealing with areas in which 
Warner was a pioneer. The first paper, by Fienberg and Jazairi, summarizes Warner’s work in the 
area of statistically balanced information technology, in which the goal is to develop statistical 
procedures to ensure that different positions regarding a policy or an issue are fairly and adequately 
represented in a debate or decision. The other two papers, by Bellhouse, and by Mangat, ef al/., are 
in the area of randomized response. The paper by Bellhouse begins with an overview of Warner’s 
contributions to randomized response and then discusses the problem of estimation of a correlation 
coefficient using data from a randomized response survey. Three randomized response setups are 
considered: unrelated question, additive constant, and multiplicative constant. The paper by Mangat 
et al. compares the efficiencies of with and without replacement sampling in the context of randomized 
response. The papers by Fienberg and Jazairi and by Bellhouse are based on presentations given 
at a special session in memory of Warner at the meetings of the Statistical Society of Canada in 
Banff in 1994. 

The next two papers, by Lavallée and by Kalton and Brick, discuss weighting schemes for cross- 
sectional estimation in panel surveys. 

Lavallée presents the Weight Share Method, used in cross-sectional estimation for longitudinal 
surveys, in a more general context. He demonstrates the unbiasedness of the method and obtains 
a general expression for the variance of the estimator of a total. Then the author illustrates the method 
by applying it in the context of the Survey of Labour and Income Dynamics of Statistics Canada. 
The estimation of variance is also discussed. 

Kalton and Brick describe weighting schemes for cross-sectional analysis of later waves of a 
household panel survey using data for all households for whom data are collected. These weighting 
schemes can accommodate new entrants to the population who move in to live with members 
of the original population, but not other new entrants. The authors discusses cases where the 
schemes are optimal as well as further weighting adjustments to compensate for nonresponse and 
noncoverage. 

Small area techniques for estimation of net undercoverage of persons in population censuses 
are discussed by Dick in the Canadian context and by Kim, Zaslavsky and Blodgett in the U.S. 
context. 

The paper by Dick describes modelling that was done in order to produce estimates of census 
net undercoverage of persons within age-sex-province categories for the 1991 Canadian Census 
using data from the Reverse Record Check and the Overcoverage Study. An Empirical Bayes model 
for direct estimates of adjustment factors is formulated and used to obtain smoothed estimates of 
those adjustment factors. The smoothed estimates of net missed persons are then raked to match 
the direct estimates of national age-sex group and provincial totals, which are considered to be of 
good quality. 

Kim, Zaslavsky and Blodgett describe the two analyses performed to test the ‘‘synthetic assumption’”’ 
of homogeneity of undercount rates between parts of different states falling in the same poststratum 
for the 1990 U.S. Census. In the first analysis, the distributions of five ‘‘surrogate variables’’ that, 
like undercount, were related to the census-taking process, were investigated using a large extract 
from the census. In the second analysis, the distribution of undercount was analyzed using the Post 
Enumeration Survey data. 

Breidt presents Markov chain designs for one-per-stratum sampling which includes systematic 
sampling, stratified simple random sampling and balanced systematic sampling as special cases. 
He introduces new designs that are shown to be competitive, in terms of the efficiency of the 
Horvitz-Thompson estimator of a total, with standard one-per-stratum designs under a variety of 
superpopulation models. Theoretical and numerical comparisons are provided. 


In This Issue 


Meeden considers the problem of estimation of the median when an auxiliary variable is available. 
He uses a non-informative Bayesian approach based on a Polya posterior for the ratios of the 
variable of interest to the covariate. The resulting estimator is empirically compared to a number 
of alternatives in terms of bias and average absolute error for a variety of real and synthetic 
populations. The Polya posterior is also to be used to generate interval estimates which are evaluated 
empirically. Robustness of the procedure to moderate departures from the assumptions is also 
considered. 

Hulliger develops design-based M-estimators for samples with unequal inclusion probabilities. 
He expresses the Horvitz-Thompson (HT) estimator as a least square functional and then makes 
it robust against outliers through M-estimators, analogous to the robustification of least square 
estimators in linear models for infinite populations. He also provides an approximation to the 
sampling variance of this robustified HT-estimator and its estimate. The results of the Monte-Carlo 
study confirm that the robustified HT-estimators outperform the HT-estimator in many outlier 
situations. 

Iachan and Kemp describe the sampling designs for two visitor sample surveys of recreational 
users of parks, a survey of National Park Service area users over a one year period, and a survey 
of users of three river basin in the Pittsburgh area. The potential problems associated with sampling 
in both time and space are described, and the ways in which the designs of these two surveys meet 
these challenges are compared and contrasted. 
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Stanley L. Warner 


1928-1992 


Born and educated in the United States, Stanley Warner 
received a Ph.D. in Economics from Northwestern 
University in 1961. In 1971, he moved to Canada, where 
he was to pursue the rest of his academic career as professor 
at York University in the Department of Economics and 
the Faculty of Administrative Studies. He died suddenly 
in August 1992 at the age of 63. 

Stan was a highly original thinker. His statistical 
research was guided by relevance and common sense. In 
his memory, a session was organized at the Statistical 
Society of Canada meetings at Banff in 1994. Two of the 
papers which follow were presented at that occasion, 
namely, David Bellhouse: ‘‘Estimation of Correlation in 
Randomized Response’”’ and Stephen E. Fienberg and Nuri 
Jazairi: ‘‘Stanley Warner’s Contributions to Statistically 
Balanced Information Technology.’’ They deal with two 
areas where Stan made pathbreaking contributions. 

Stan Warner is known to many, especially to survey 
statisticians, as the man who invented randomized response. 
This technique is used in surveys with sensitive questions 
as in a survey concerning drug usage among high school 
students. The objective is to eliminate two embarrassing 
nonsampling errors: measurement error and nonresponse. 
Bias caused by such errors is a problem that has haunted 
statisticians since the beginnings of survey taking. Standard 
methods exist to ‘‘adjust for’’ such bias. They may reduce 
but do not eliminate the bias. Stan, disregarding the 
conventional wisdom, tackled the problem in a completely 
unorthodox way and came up with an unbiased solution 
useful at least for some surveys. His seminal article, 
‘‘Randomized Response: A Survey Technique for Elimi- 
nating Evasive Answer Bias’’, appeared in the Journal of 
the American Statistical Association in 1965. 

Randomized response, if carried out according to the 
intentions, guarantees the anonymity of the respondent: 
a ‘‘yes’’ answer will not identify the respondent since his 
or her question is selected at random. Unbiased estimation 
of the ‘‘yes’’ proportion in the population is nevertheless 
possible (usually at the price of some increase in variance) 
because the survey taker knows the number of ‘‘yes’’ 
responses as well as the probability with which the random 
choice device selects the sensitive question (rather than its 
opposite or a completely unrelated question). 

The idea struck the imagination of many statisticians. 
Numerous modifications, refinements and extensions were 
given, as evidenced through the 1988 bibliography of 
contributions to randomized response put together by 
Chaudhuri and Mukerjee. Why this stream of papers? 


That nonresponse bias was always an important practical 
problem without a satisfactory solution is not the whole 
explanation. There is also the fact that it seemed like 
magic, even to experienced statisticians, that valid answers 
could be obtained without knowing what question had 
been asked of the respondent. Of course now that the idea 
exists, it is not hard to explain; in fact it has become a 
favorite classroom example, useful even in an elementary 
statistics course, to show students the powers of statistical 
reasoning. 

Implementing randomized response in practice requires 
some special arrangements, including a choice device that 
randomly selects a question. As time went by, Stan realized 
that the technique had to be adapted to modern low cost 
data gathering. As late as 1989 at the International Statis- 
tical Institute meetings in Paris, he presented a ‘‘quick 
randomized response”’ version suitable for telephone surveys 
and touch tone entry, thereby reducing time and cost. 

The paper by David Bellhouse traces the development 
of randomized response in more detail. 

Later, Stan’s interest focused on statistical procedures 
for balanced information, which occupied him from about 
1975 and on. In the last few years of his life he was working 
on a book on the topic; the manuscript is now being 
prepared for publication. 

The goal of balanced information technology is to give 
statistical procedures to ensure that different positions 
regarding a policy or an issue can be fairly and adequately 
represented. Stan’s first paper on this topic, entitled 
““Advocate Scoring for Unbiased Information’’, appeared 
in the Journal of the American Statistical Association in 
1975. It deals with the situation in which advocates are 
each to give pro and con information regarding an issue 
to a number of individuals who, after exposure to the 
information, are to express opinions for or against the 
issue. Each advocate is charged with using the given data 
to prepare separate pro and con cases. 

Decisions are frequently made on information provided 
by advocates; this occurs in government, education, law, 
etc. One can imagine that Stan was concerned with the 
incomplete and sometimes arbitrary way in which quanti- 
tative information is used in decision making of the utmost 
importance, including political summits, where prestige, 
mistrust and political consideration would often prevent 
the parties from meeting on even ground. 

The paper by Stephen E. Fienberg and Nuri Jazairi 
presents the main ideas of this work, which is probably less 
familiar to statisticians. 


Another example of Stan’s creative spirit is the fact that 
he developed, with his wife, a musician, a system for music 
notation which is in wide use. 

I did not know Stan Warner at the time of his original 
work on randomized response but keep vivid memories of 
conversations with him later in his career. These occasions 
were not that numerous; however, each left a strong 
impression on me. The warmth, modesty and unassuming 


Stanley L. Warner 


manner of this highly original man could not fail to make 
an impression. Stan was a real scholar, not a run-of-the- 
mill researcher: he did not hesitate to follow the sometimes 
lonely route laid out to him by belief in ideas that were 
truly his own. 


C.-E. Sarndal 
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Stanley Warner’s Contributions to Statistically Balanced 
Information Technology 


STEPHEN E. FIENBERG and NURI JAZAIRI! 


ABSTRACT 


Stanley Warner was widely known for the creation of the randomized response technique for asking sensitive questions 
in surveys. Over almost two decades he also formulated and developed statistical methodology for another problem, 
that of deriving balanced information in advocacy settings so that both positions regarding a policy issue can be 
fairly and adequately represented. We review this work, including two survey applications implemented by Warner 
in which he applied the methodology, and we set the ideas into the context of current methodological thinking. 


KEY WORDS: Advocate scoring; Bayes’ Theorem; Embedded experiment; Logistic regression; Survey analysis. 


1. INTRODUCTION 


Consider some recent controversial public or professional 
issues such as: 


1. Should Canada endorse the North American Free 
Trade Agreement? 


2. Should Quebec secede from the Canadian Federation? 


3. Should the American Statistical Association adopt a 
program to certify statisticians? 


4. Should smoking be banned in all restaurants in Ottawa? 


The discussions and debates surrounding such issues 
often reflect highly polarized positions and ‘‘pro’’ and 
“‘con’’ arguments can strongly influence the opinions of 
individuals in the relevant populations of interest (e.g., 
Canadian residents, ASA members, those who frequent 
restaurants in Ottawa). How to think about the presentation 
of such advocacy information in a balanced fashion is the 
topic of this paper. 

It has often been said that only a small fraction of scien- 
tists make a truly novel research contribution once in their 
lifetime. Far fewer are responsible for multiple innovations. 
Stanley Warner is well-known for his creation and 
development of the randomized response model for surveys 
and that contribution has been widely hailed as a major 
development in statistics. What is less well known is his 
truly novel approach to the problem of balanced informa- 
tion in advocacy settings, on which he worked over a 
period of almost 20 years. As York University colleagues 
of Warner’s at the time of his death in 1992, we know how 
seriously he took the obligation of statistics and statisticians 
to deal with such complex problems, and this work is one 
example of how he attempted to fulfill the obligation. 

Our goal in this paper is to reintroduce Warner’s ideas 
on the topic of balanced information in advocacy settings 


to the profession and to demonstrate how they fit into 
current survey practice and methodological thinking. In 
Section 2, we present his basic approach to the advocacy 
problem and we describe the statistical model he chose to 
focus upon (Warner 1975). In Sections 3 and 4, we discuss 
embellishments of the basic approach which he presented 
in subsequent papers (e.g., see Warner 1981, 1984, 1985, 
1987a), and we end by describing how Warner continued 
to pursue this research program up until the time of his 
death. In the process, we also stress the importance Warner 
attached to the application of his ideas. 


2. THE BASIC PROBLEM 


In a thoughtful, well-argued, yet provocative 1975 
paper in the Journal of the American Statistical Association, 
Stanley Warner first presented the issue of measuring the 
impact of advocacy and balance on public opinion in 
connection with controversial issues. He did so by asking 
(and then answering) a pair of interrelated questions: 


1. How can we estimate what the population would 
conclude on issue were each of the members 
provided with balanced information on the topic? 


Warner’s idea for answering this question was to use 
advocates to present summaries of arguments, both “‘pro”’ 
and ‘‘con,’’ and to implement this in a factorial experi- 
mental design to different samples, and in the process 
achieve information about a balanced presentation. This 
then leads rather naturally to the second question: 


2. How can we rate or score advocates in such settings? 


He developed his formulation to answer the two questions 
simultaneously and, in doing so, he used both economical 
and statistical arguments. In this paper, we focus on the 
statistical portion of his arguments and refer the interested 
reader to Warner’s paper for the economic details. 


! Stephen E. Fienberg, Department of Statistics, Carnegie Mellon University, Pittsburgh, PA 15213, U.S.A.; Nuri Jazairi, Department of Economics, 


York University, North York, Ontario, Canada, M3J 1P3. 
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Consider a pair of advocates or advocate teams whose 
role it is to brief individuals on the arguments associated 
with a controversial issue, H. Let P(H) and P(A) denote 
the proportion of the number of subjects in a given 
population ‘‘for’’ and ‘‘against’’ issue H. Let F; and A; 
denote ‘‘pro’”’ and ‘‘con’’ presentations of advocates j and 
j, respectively for i, j = 1, 2. Let P(H | Fj, A;) and 
P(H | F;, A;) denote the number of subjects ‘“‘for’’ and 
‘‘against’’ issue H after hearing ‘‘pro’’ case from advocate 
i and ‘‘con’’ case from advocate /. 


Warner defined the ‘‘net information’’ associated with 
F; and Aj as 


I(F;,Aj) = In[P(H | F;,A;)/P(A | F;,Aj)] 
— In{P(H)/P(A)}. (1) 


Formula (1) is, of course, the logarithm of the Bayes 
factor, or what Good (1950) called the weight of evidence. 
While Warner recognized the evocative nature of the use 
of Bayes’ Theorem here, his approach towards its use was 
purely frequentist. 

Similarly, Warner defined the net information associated 
with “‘pro”’ and ‘‘con’’ cases of F; and A;, separately: 


I(F;) = In{P(A | F))/P(A | F))] — In[ P(A) /P(A)), 
(2) 


1(A;) = In(P(A | Aj)/P(A | A;)] — In[P(H)/P(A)). 
(3) 


The simplest assumption we can make relating the joint 
and marginal information quantities is that of independence 
of the ‘‘pro’’ and ‘‘con’’ cases, 


DE ieee ae Le (4) 


fori = 1, 2, andj = 1, 2. This assumption allows for 
some direct comparisons and, as we shall see, can be 
checked empirically. 

In order to ensure that the advocates fairly treat both 
““‘pro”’ and ‘‘con’’ positions, Warner proposed to reward 
them on the basis of the sum of the net information they 
provided, i.e. 


I(F;) + I(A)). (5) 


Economic theory, Warner argued, suggests that rewarding 
advocates in this fashion will lead them to at least strive 
to approximate the ‘‘unbiased information’’ associated 
with maximization under resource constraints. Thus we 
need to estimate the quantity in expression (5) along with 
the posterior odds implied by unbiased information: 


PCD ere) aC si ae ai (6) 


‘*Balance’’ in design for data collection was the key to 
Warner’s plan for estimation. 

Warner’s estimation plan was linked to his application. 
The controversial issue was the completion of the north- 
south Spadina Expressway in Toronto (Warner’s home 
city). The original first section of the expressway was con- 
structed in 1966 and, after much debate, the remainder of 
the project was canceled in 1971. Two years later, in 1973, 
Warner conducted a survey to learn what proportions of 
the population of registered voters of Metropolitan Toronto 
were for or against the original expressway plan. He took 
a random sample of 1,360 registered voters (1% of the 
corresponding population) divided into 8 equal subsamples 
of size 170. Two advocate teams prepared written positions, 
both ‘‘pro’’ and ‘‘con’’ the expressway, and one of each 
was included in the mailing. The order of presentation of 
the two written positions was also varied producing a 
2 x 2 x 2 experimental design with the first variable 
corresponding to who prepared the ‘‘pro’’ brief, the second 
to who prepared the ‘‘con’’ brief, and the third to the order 
of presentation (‘‘pro’’ first or ‘‘con’’ first). Advocates 
were paid a basic fee and a larger amount was set aside to 
be paid to the team with the ‘‘best combined score.’’ This 
is an excellent example of a factorial experiment embedded 
within a survey, and fits well with the spirit of embedding 
described in Fienberg and Tanur (1988). 


Table 1 


Sample Preferences for Spadina Expressway After 
Information by Advocates 


Sample i j k For Against Undecided Total pjjp  nijx 
1 eval Tal ae 4 1 2 ee SAG 26 
2 ile ea 9 2 29.666 = 27 
3 Py GS 8 0 34.764 34 
4 2 ay a) 11 1 33056) 32 
5 Zola lia 28 10 1 SEY SIRS Ait 
6 Dole 2a 11 1 AS weEdde 25 
a Dip ae “ily 9 SIG) 16 1 36) 5.5425 © 35 
8 Pie Pop PR NE) 7 2 213 ges! oe BO 


Source: Warner (1975). 


In the cover letter, Warner asked respondents to return 
prepaid postcards indicating their preferences after reviewing 
the briefs. At the cut off date, 262 cards had been returned 
for a response rate of about 20%. The resulting data, in 
Table 1, are reproduced from Warner (1975). 

Let pjjx be the true proportion of the population ‘‘for”’ 
the expressway, in group (i, /, k). Then, with an additive 
term for order of presentation, the model of expression 
(1) becomes 


In( pijg/ (A — pix) = InLP(A)/P(A)) 


cup A Oa) Nace Me: rg Bee a 8 IB 


Survey Methodology, June 1995 


We now recognize expression (7) as a linear logit model, 
and the sampling scheme as product-binomial (ignoring 
the correction for the 0.2% sampling fraction). Of course 
when Warner did this work it preceded the existence of a 
monograph by Bishop, Fienberg, and Holland (1975), and 
was virtually concurrent with Nelder and Wedderburn’s 
(1972) paper on generalized linear models. Thus his paper 
made no reference to the now extensive literature on logit 
and loglinear models. 

To estimate the parameters in expression (7), Warner 
used weighted least squares, which yields both estimated 
coefficients and standard errors. Instead of dealing directly 
with the parameters in expression (7), he redefined them, 
in part to simplify computation and in part to aid in their 
interpretation: 


Pe PCriyee tl (Py) = TCAs) i I(F,) — I(A2) 
P(A) 2 2 
oj Aa? (8) 
2, 
B. = I(F\) — I(F)), (9) 
B; = I(A,) — I(A), (10) 
B, = D, — Dy. (11) 


The coefficient 6; is an ‘‘intercept’’ or normalizing para- 
meter, while 65, 83,and 6» + (3; measure the performance of 
the advocate teams, and 6, measures the order effect. The 
net information provided by team 1 is 8; + .5(8, — 83), 
and that provided by team 2 is 6, — .5(6, — 63). The 
difference in net influence is thus 8, — 3. 


Table 2 
Weighted Least Squares Estimates of Theoretical 
Parameters 

Parameter Estimate oe 

Std. Error 
By Shey alley) 
B> .648 277 
B3 — .383 a5 
B4 .528 .274 
Bo + B3 .264 .386 
By + .5B2 — .583 1.228 .266 
B; — -5B2 + -583 .196 PANS 
Bo — B3 1.032 395 


Source: Warner (1975). 


We reproduce Warner’s estimation results in Table 2. 
We have double-checked the estimated values in Table 2 
using the generalized model routines in S+ , which utilize 
a version of iteratively weighted least squares (maximum 


likelihood in this case). Our logit model computations agree 
with Warner’s to two decimal places. The residual deviance 
for this model equals 1.95 with 4 d.f., indicative of a 
remarkably good fit and offering strong support for the 
reasonableness of the independence assumption of expression 
(4). 

In interpreting the results in Table 2, Warner noted that 
his economic analysis leads to the conclusion that the overall 
proportion of the population in favour of H when presented 
with unbiased information lies between the ‘‘pure’’ estimates 
for the 2 advocate teams, or in the present instances (.55, 
.77). These bounds correspond to the estimates in the 2nd 
and 3rd last lines of the table. As was clear from Table 1, 
no matter how we combine ‘‘pro’’ and ‘‘con’’ arguments, 
the majority in each subgroup favored completion of the 
expressway. Warner observed that we might be tempted 
to use By to produce a ‘‘best estimate’’ of the value of p 
corresponding to unbiased information, but he argued for 
a higher value, since Team | is superior to Team 2 in terms 
of total information, /.e., B> - B3 > 0. (The superiority 
of Team | is quite evident from a quick examination of 
Table 1 and does not require the full analysis.) 

Warner ended his 1975 paper by pointing out all of the 
shortcomings of his small experiment, and his initial 
modelling efforts. What we can observe in retrospect is the 
way in which he was able to attack a very complicated public 
policy and survey problem using a simple but ingenuous 
model, as well as a rigorous estimation scheme built on the 
solid framework of a factorial experiment embedded in 
a sample survey, and then actually applying the method- 
ology to produce an answer for a real problem. 

It is worth noting that the first version of this paper was 
submitted for publication to JASA in June 1972, before 
Warner had actually carried out the empirical study on the 
Spadina Expressway controversy. Over two years passed 
before he resubmitted a revised version of the paper with 
the detailed example. Even well-known authors with inno- 
vative ideas often struggle to have their work published 
in major statistical journals, and a compelling empirical 
application is always of help. 


3. EXTENSIONS AND A SECOND APPLICATION 


Warner extended his balanced information approach 
in a second paper (Warner 1981), focusing on yet another 
application. This paper also signals a substantial change 
in Warner’s thinking about statistics and probability, towards 
a subjective Bayesian approach and away from the classical 
approaches that he stressed in his early career. While the 
reported analyses are still frequentist in nature, Warner 
used, at least informally, the assessments of prior pro- 
babilities in a manner that fits rather naturally with the Baye- 
sian formulation of expression (1) above. 


10 Fienberg and Jazairi: Stanley Warner’s Contributions to Statistically Balanced Information Technology 


In March 1972, the Canadian Federal Government 
announced a plan to build a second Toronto International 
Airport to the east of the city in Pickering, Ontario. This 
led to considerable controversy. In 1974, the government 
appointed a 3-person commission of inquiry. Warner 
carried out a concurrent but independent survey experiment. 
The question he posed was whether or not the Pickering 
Airport should be built before the year 2000. The general 
structure of the experiment was similar to the previous one 
on the Spadina Expressway controversy, but with some 
differences: 


(i) This time his study population was economists. 


(ii) He incorporated 2 ‘‘neutral’’ control sub-samples, 
which received neither ‘‘pro’’ nor ‘‘con’’ statements. 


(iii) Respondents in the 8 experimental subsamples gave 
probability assessments (instead of 0-1 values) after 
assessing the advocacy positions. Those in the control 
groups also gave their probability assessment. 


The test population was limited to those economists 
who belonged to the Canadian Economic Association or 
who could be identified as professors or lecturers in an 
economics department in a Canadian university. The 
survey was done via mail in two stages - the first identified 
those willing to read detailed briefs and ‘‘report opinions 
regarding an undisclosed federal project,’’ and the second 
mailing divided those willing to participate into 10 sub- 
samples, corresponding to the 2 x 2 x 2 design of 
Section 2 plus the 2 control samples consistent of those 
who were asked for their opinions without briefs. A total 
of 726 economists participated in the experiment. In 
Table 3, we provide Warner’s summary of the data for the 
8 experimental subsamples in which he aggregated the 
posterior judgments into three groups according to whether 
they were substantially greater, nearly equal, or substan- 
tially less than 0.5. The data have been further aggregated 
across the 8 experiment groups. The results have been post- 
stratified according to whether the economists were 
professors, graduate students, or others. The data on 
‘‘prior beliefs’? come from a combination of the two 
control groups. 


Table 3 


Test Population Opinions on Pickering Airport 


Professors Students Others Totals 


Before After Before After Before After Before After 
Briefs Briefs Briefs Briefs Briefs Briefs Briefs Briefs 


For 9 58 9 32 11 71 29 161 
(.143) (.266) (.257) (.288) (.180) (.298) (.182) (.284) 
Against 32 155 12 a2 36 160 80 387 
(.508) (.711) (.343) (.648) (.590) (.672) (.503) (.683) 
Undecided IY) 5 14 7 14 7 50 19 
(.349) (.023) (.400) (.063) (.230) (.029) (315) (.033) 
Totals 63 218 35 111 61 238 159 567 


(1.000) (1.000) (1.000) (1.000) (1.000) (1.000) (1.000) (1.000) 
Source: Warner (1981). 


Note that all three groups had substantial negative 
opinions about the proposed airport, a posteriori, and 
that the differences in proportions of undecided between 
the experimental and control subsamples provide evi- 
dence that the advocacy briefs affected public opinion 
on the issue. Warner’s formal statistical analysis of the 
data focused solely on the 8 experimental subsamples 
and utilized three variants of the formal model in expres- 
sion (9) and the reparameterization of expressions (10) 
through (13): 


(i) A logit structure similar to that in Warner (1975) 
based on the aggregation in Table 3, with ‘‘undecideds”’ 
in effect imputed as belonging in either the ‘‘pro”’ or 
‘*con’’ categories with probability 0.5. He called this 
a Simple Aggregate Influence model. 


(ii) A more direct approach, which averaged the posterior 
assessments to get ‘‘aggregate proportions’’ in favor, 
and then treated these observed proportions as if they 
were binomial. He called this a Weighted Aggregate 
Influence model. 


(iii) A two-stage model, which first used individual-level 
assessments breaking up the range of 0 to 1 into 17 
levels, and then a ‘‘variable coefficient’’ regression 
model analysis. He referred to this as an Average 
Disaggregate Influence model. 


Each analysis involved the use of a different form of 
weighted least squares to estimate the coefficients of 
interest. 


In Table 4, we provide Warner’s estimated coefficients 
under all three models and analyses. The results are similar 
across models and we can summarize the findings as 
follows: 


(a) Team 2 clearly presented the strongest case (B> and B3 
are both positive and similar for all three columns). 


(b) The estimated aggregate influence for Team 2 is 
[B; + PSi(B5 - B3)] = —(0.688 corresponding to an 
estimated proportion in favor of the airport project of 
j eh RE DE 


(c) The disaggregate influence for Team 2 corresponds to 
an estimated proportion in favor of the airport project 
Giip = 0.35). 


(d) The effect of order of presentation (B4) suggests that 
the brief appearing first in the enclosures had greater 
impact, and is consistent with the hypothesis that the 
‘*previous information favoring one position serves to 
discount new information against that position.”’ 


It turns out that the advisor for Team 1 felt that construc- 
tion of the airport could not be defended and this seriously 
handicapped the ‘‘pro’’ efforts of Team 1 (something 
reflected in the estimates of 6). 
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Table 4 
Estimated Case Influence for Pickering Airport 
Experiment 
Simple Weighted Average 
Parameters Aggregate Aggregate Disaggregate 
Influence Influence Influence 
By — .857 — 529 — .736 
(.093) (.047) (.065) 
Bo .485 £3377) .462 
(.188) (.097) (.132) 
B3 .147 .146 .187 
(.188) (.097) (.132) 
Ba 313 .209 307 
(.186) (.095) (.129) 
N 8 8 567 


Source: Warner (1981). 


4. OVERLAPPING INFORMATION 


Warner worried that the information used in the 
Pickering Airport survey experiment involved an overlap 
between the ‘‘pro”’ and ‘‘con’’ cases, and there was also 
an overlap between the prior information available to the 
respondents and that presented by the advocates. He turned 
to this question several years later in Warner (1984, 1985), 
using a formal argument drawn from sampling theory. 

Warner’s idea was to consider N pieces of independent 
information being used to influence the proposition in 
question. Let Yj; be the information content seen by the 
i-th person in the j-th piece of information. Then the prior 
odds for the i-th individual is 


Wipe Dilla, ree’ ys (12) 
J€EA(i) 


where A (7) is the collection of information seen by the i-th 
person prior to the presentation of the advocacy argu- 
ments. If A (7) is empty, the initial log-odds for individual 
i should be 0 and thus p; = 0.5. 

The presented ‘‘pro’’ and ‘‘con’’ summaries draw on 
a subset, S, of m out of the N units. Suppose that partici- 
pants act ‘‘rationally’’ and are not further influenced by 
data which has been seen before. The added information 
is then 


Dy eitgn sie) ein tg (13) 


JES JEA(I) NS 


If the m units of information are randomly selected 
without replacement from the total N, then this implies 
that we can treat the units in A (7) MS as having been selected 
at random without replacement. We can treat the informa- 
tion from these overlapping units as following a hyper- 
geometric distribution, and then we rewrite expression (1) as 


4 


where /; is the net information in the summary for the i-th 
individual, S; is the average of the Y;;’s for those data 
units j€S, and the error term € has zero conditional 
expectation, /.e., 


Efé | [p;/(1 — pj)l} = 0. (15) 


If we group subjects in an advocacy experiment exposed 
to the same ‘“‘pro’’ and ‘‘con’’ briefs according to the 
values of p;, then differences in net information should 
be related to [p;/(1 — p;)] according to equation (14). 
Warner (1984) did this with additional simplifying assump- 
tions and then analyzed the data from the Pickering experi- 
ment using assessments, the control groups and the Team 2 
‘‘pro’”’ and ‘‘con”’ group, aggregated according to whether 
the respondent was a professor, a student, or other. The 
problem with using the data from the Pickering experiment 
is that we are in effect matching the individuals in the 
control group and the experimental group. What we really 
want is both the prior and the posterior assessments from 
the same individual (Warner 1987b). 

Warner (1985, 1987a) returned to this theme of over- 
lapping information, and he extended the model of expres- 
sion (14) to take the form: 


Ke ieaS pete iry (Zinthy)s (16) 


where we have in effect replaced the coefficient —m/N 
in expression (14) by D;r;, where D; = — 1 isa discount 
factor and E(r;) = m/N. He then showed how to estimate 
the coefficients in this ‘‘random coefficients’’ regression 
model using generalized least squares, under various 
assumptions about the correlations among the quantities 
in (16). 

He then applied the approach to a new set of data 
collected for a telephone survey of Carleton University 
students on the question of whether an elected Canadian 
Senate would be preferable to the existing appointed 
Senate. The interviewees were asked for their opinion on 
the issue expressed as a probability. They were then 
presented with a 6-sentence summary of a television debate 
on the topic, and asked to reconsider their probability 
assessments. Of the 417 participants, 316 gave prior prob- 
abilities different from 0 and 1. Of this group, 163 actually 
changed their assessment, and overall the average log-odds 
after the summary was virtually the same as it was before, 
but with a slightly smaller variance. Warner actually fit 
the model to the data, and the fitted equations were con- 
sistent with the notion of partial discounting of the infor- 
mation that they had already seen. 

This was essentially Warner’s last published contribution 
to the topic of balanced information assessment. At the 
time of his death, Warner was hard at work on a book-length 
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manuscript whose title, Statistically Balanced Information 
Technology, suggests that he was attempting to synthesize 
and extend his ideas on the topic. Unfortunately, we 
have only been able to locate the early chapters of this 
book and these include just the introductory ideas on 
probability and regression that he expected to utilize in the 
later chapters. 


5. FURTHER OBSERVATIONS 


Warner’s balanced information technology addresses 
the common problem of adversarial policy advocacy which 
may give rise to confusion and incorrect decisions because 
of imbalance in the presentation of the relevant facts. 
Examples of how the adversarial approach to dispute 
resolution in a legal setting could have distorting effects 
on questions of scientific fact are discussed in Fienberg 
(1989; see especially Appendix H by Vidmar). Among the 
responses to this situation have been repeated proposals 
to establish a science court to ensure balance in organizing 
the information relevant to a factual dispute and reaching 
decisions. In these proposals, the science court itself is an 
adversarial system, but based on well-defined procedures 
for the selection of issues, advocates, and judges designed 
to ensure impartiality and minimize the effects of personal 
bias. Warner’s approach outlined here is a formal way to 
achieve precisely this kind of impartial result. 

Warner’s progression through the various stages of the 
work on balanced information was paralleled by a shift 
in his outlook on the foundations of statistics. He was 
trained as an economist and a classical statistician and his 
early statistical contributions, including the work on 
randomized response models, were all set in a frequentist 
statistical framework. The 1975 paper on advocate scoring 
represented his first step towards a subjectivist perspective 
and, with each successive paper, he added further elements 
of the Bayesian approach. In Warner (1979), Stan articu- 
lated this shift in thinking and it is especially apparent in 
the early chapters of his unpublished book. At the May 
1992 annual meeting of the Statistical Society of Canada 
in Edmonton, his last public lecture, Stan described devices 
for the solicitation of probabilities that he had been 
developing for the book. 

We can only speculate about how Stan’s subjectivist 
synthesis of balanced information technology would have 
looked had he been able to complete the book. But given 
the depth of his commitment to the Bayesian approach and 
its recent methodological innovations, we expect that it 
would have included a hierarchical generalized linear 
model approach and utilized the latest developments in 
Markov Chain Monte Carlo simulation techniques. 

Stanley Warner was constantly using the ideas from his 
research in the classroom and in reflecting back upon the 
work described here, he noted: 


‘*.. almost all of the basic elements of an elemen- 
tary statistics course are to some degree represented 
in these procedures, and the problems in modeling 
and design that are suggested could be considered 
at quite an advanced level’’, (Warner 1987b). 


The statistics profession has lost a true innovator and 
a great educator. We count ourselves amongst Stanley 
Warner’s students and we continue to learn from his work. 
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Estimation of Correlation in Randomized Response 


D.R. BELLHOUSE! 


ABSTRACT 


Stanley Warner’s contributions to randomized response are reviewed. Following this review, a linear model, based 
on random permutation models, is developed to include many known randomized response designs as special cases. 
Under this model optimal estimators for finite population variances and covariances are obtained within a general 
class of quadratic design-unbiased estimators. From these results an estimator of the finite population correlation 
is obtained. Three randomized response designs are examined in particular: (i) the unrelated questions model of 
Greenberg et al. (1969); (ii) the additive constants model of Pollock and Bek (1976); and (iii) the multiplicative 
constants model of Pollock and Bek (1976). Simple models for response bias are presented to illustrate the effect 


of this bias on estimation of the correlation. 


KEY WORDS: Additive constants model; Linear models; Multiplicative constants model; Response bias; Unrelated 


question model; Variance estimation. 


1. A BRIEF OVERVIEW OF WARNER’S 
CONTRIBUTIONS TO 
RANDOMIZED RESPONSE 


Randomized response is a technique used to elicit 
responses to sensitive questions. It was developed thirty 
years ago by Stanley Warner (Warner 1965) to estimate 
a proportion under a simple random sampling design with 
replacement. The development was a substantial intellec- 
tual achievement requiring much originality of thought. 
How does one get truthful responses to sensitive questions? 
Warner’s solution was to get the response without the 
interviewer knowing whether the sensitive question had 
actually been asked. He devised the probabilistic structure 
to the questioning so that an estimate of the required 
proportion could be obtained. In Warner’s original for- 
mulation the population is divided into two mutually 
exclusive and exhaustive groups, A and B. It is of interest 
to estimate the proportion z of the population belonging 
to group A. To do this, a spinner is constructed with a face 
marked with the letters A and B. The construction is such 
that the spinner points to the letter A with probability p 
and to B with probability 1 — p. The interviewee spins the 
spinner and is required only to say yes or no according to 
whether or not the spinner points to the interviewee’s 
correct membership group. The with replacement design 
allows estimation of 7 by maximum likelihood. 

This very original idea has received substantial attention 
over the past thirty years. Since Warner’s original work, 
several randomized response techniques have been suggested 
for the estimation of a proportion or set of proportions 
as in polytomous data, or for the estimation of a popula- 
tion mean with continuous data. A variation on Warner’s 
original theme is asking the sensitive question or an 


unrelated question with probabilities p and 1 — prespec- 
tively. This was originally due to Greenberg et a/. (1969). 
Other variations with continuous data include adding a 
random variable to the response to the sensitive question 
or multiplying the response by a random variable. The 
underlying theme to any of these techniques is the masking 
of the original response in such a way that the sensitive 
information cannot be attributed to any single respondent 
but that information on the sensitive attribute can be 
extracted from the whole sample. A substantial literature, 
including a monograph by Chaudhuri and Mukerjee (1988), 
has grown up around these techniques. Nathan (1988) has 
provided a fairly comprehensive bibliography of this 
literature. Umesh and Peterson (1991) have given several 
detailed examples from very diverse areas of the applica- 
tion and applicability of the techniques of randomized 
response. 

With several different randomized response techniques, 
the question arises as to how to compare the different 
methods. Minimization of variance cannot be the sole 
criterion. Each method is designed to protect the privacy 
of the respondent. A gain in efficiency, in terms of 
variance, by the choice of different values of the proba- 
bilities in the randomizing device, or by the choice of one 
randomized response method over another, could lead to 
jeopardizing the privacy of the respondents. In response 
to this, Leysieffer and Warner (1976) and Warner (1976) 
formulated natural measures of respondent jeopardy. 
These measures are related to the probability of the inter- 
viewer being able to infer the interviewee’s response to the 
sensitive attribute. The theory of respondent jeopardy is 
reviewed in Chaudhuri and Mukerjee (1988) and some 
practical considerations regarding respondent jeopardy 
are reviewed in Umesh and Peterson (1991). 


' D.R. Bellhouse, Department of Statistical and Actuarial Sciences, University of Western Ontario, London, Ontario, N6A 5B7. 
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Stanley Warner made two other contributions to the 
literature of randomized response. The first contribution 
is directly related to the results obtained here. With the 
explosion of new ideas and new techniques in randomized 
response, Warner (1971) formulated a linear model which 
unified the theory. Most of the randomized response 
techniques at that time could be put in his linear model 
framework. The second contribution was in response to 
the growing use of telephone interviewing. Stem and 
Steinhorst (1984) described randomized response methods 
applicable to telephone interviewing and to mail question- 
naires. Warner (1986) suggested practical natural random- 
izing devices, such as the serial numbers on paper money, 
for use in telephone interviewing. 

The major topics in randomized response methodology 
are: the development of randomized response techniques, 
the comparison of these techniques through the concept 
of respondent jeopardy, the construction of reasonable 
randomizing devices, the development of a unified theory 
of randomized response, and the validation of randomized 
response techniques through field studies. Stanley Warner’s 
contributions to randomized response touch on most of 
these major developments in the subject. Moreover, most 
of these contributions were substantial and influential. He 
is the originator of the technique. His original setup of a 
dichotomous population was quickly generalized to a 
polytomous one and to populations with continuous 
measurement. New randomized response techniques 
continue to be developed. Warner was at the forefront of 
evaluating randomized response designs through the 
modeling of respondent jeopardy. His work in the devel- 
opment of a unified linear model for randomized response 
designs was the foundation on which a unified theory of 
randomized response has been built. 


2. INTRODUCTION TO ESTIMATION 
OF CORRELATION 


Consider a finite population of size N with two mea- 
surements of interest x; and y,;forj = 1, ..., N. It is of 
interest to estimate the finite population correlation 


—— 
Ox Oy 
where ox, = Y (x; — X)(y¥; — Y)/Nis the finite popu- 
lation covariance between the variables x and y and where 
oy and o; are the finite population variances of the vari- 
ables x and y respectively. To estimate p a sample of fixed 
size n is chosen with probability P(s) from the finite 
population where s denotes the set of finite population 
units chosen for the sample. The expectation operator with 
respect to the sampling design P(s) is denoted by E,. 
Estimators for p are obtained by replacing 07, 07 and oy, 


by their respective estimators, unbiased or biased, optimal 
in some sense or otherwise. 

To illustrate the general results obtained here for esti- 
mation of the finite population correlation coefficient, 
three particular randomized response techniques will be 
considered: 


(i) The unrelated questions model due to Greenberg et al. 
(1969). The sensitive question is asked with proba- 
bility p and an unrelated question which is not sensi- 
tive is asked with probability 1 — p. For estimation 
of the mean it is assumed that the finite population 
mean X of the unrelated question is known. For esti- 
mation of variance it is also assumed that o2 is known. 


(ii) The additive constants model due to Pollock and Bek 
(1976). The outcome of a random variable from a 
known probability distribution is added to the value 
of the response to the sensitive question. 


(ili) The multiplicative constants model due to Pollock 
and Bek (1976). The value of the response to the sensi- 
tive question is multiplied by the outcome of a random 
variable from a known probability distribution. 


Edgell et al. (1986) have provided estimators for p under 
the unrelated questions model and the additive constants 
model. 

Most randomized response designs that have been con- 
sidered have assumed that the sampling design is simple 
random sampling either with or without replacement. 
Since the results obtained here are under a fixed size 
design, the simple random sampling design assumed here 
is without replacement. 

Assume that both x and y are sensitive variables. Conse- 
quently, a randomized response technique is used to obtain 
information on both these variables. Let w; and z,, for 
J € sbe the sampled measurements that are obtained. Let 
u;and v;forj = 1, ..., N be the nonsensitive measure- 
ments associated with x; and y, respectively. Under the 
unrelated question model (randomized response model (i)) 
u; and v; are the responses to the unrelated questions for 
the j-th individual. Under the additive constants model or 
the multiplicative constants model (randomized response 
models (ii) or (iii)) u; and v, are the j-th outcomes of 
random variables from two, possibly different, known 
probability distributions. 


3. RANDOM PERMUTATION MODELS 


Several models for the finite population measurements 
have been put forward in the survey sampling literature. 
Here attention is focused on the random permutation 
models of Rao (1975) and Rao and Bellhouse (1978). 
One compelling reason for using these models is that the 
model parameters have a direct interpretation in the finite 
population of interest since model parameters in random 
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permutation models are also finite population parameters. 
In the simplest context for random permutation models 
it is assumed that the N-dimensional vector of finite 
population measurements is a random permutation of an 
N-dimensional vector of fixed numbers. Rao (1975) has 
shown how this assumption leads to a linear model. 
Bellhouse (1980) extended this model to randomized 
response designs under unequal probability sampling. 

The model and associated designs applicable to unequal 
probability sampling are not easily applicable to estimation 
of variances and covariances either with or without a ran- 
domized response. Consequently, a special case of the 
model in Bellhouse (1980) is given here. In the model which 
follows there are two different expectation operators at 
work which together yield a composite expectation E,,,. 
These expectation operators are: E,, the expectation 
operator with respect to the randomizing device, and E,,, 
the expectation operator with respect to the random per- 
mutation model. The composite expectations E,, = E,,E, 
and E = E,,E,. For the random permutation model we 
assume that the pairs (x;, yj), / = 1, ..., Narea 
random permutation of a set of N fixed pairs of numbers, 
say (p;,q;),J = 1, ...,.N. This is a special case of model 
(4.1) in Rao and Bellhouse (1978); the more general model 
in Rao and Bellhouse (1978) was used in double sampling 
and sampling on two occasions. The unrelated questions 
randomized response model (randomized response model 
(1)) requires an additional assumption that the quadruples 
(Xj, Vj. Uj, Vj), J = 1, ..., Narea random permutation 
of a set of N fixed quadruples of numbers, say (p;, j, “5 
bE eiatda~ cee. 

Assume that the randomizing device coupled with the 
random permutation model leads to the following linear 
model: 


(1) 


forj = 1,...,Nwhere X and Y are the finite population 
means of the x and y measurements respectively and where 
Ota were N, 

Em(ey) = Em(e2j) = 0, 

Em(etj)) = dion + Yor + WX + vy X?, 


Em(€5;) = b205 + Yo + WY + Yn’, 


Ey, (ey; C1x) = 5,02 ot Nps Em (22; Cox) — 5205 a ro, 


Or, Wh eana 


Eker; €>;) = $30 xy a5 V3, and 


Eke; Cx) = 630xy at \3; for ij xe Ike (2) 
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and all other higher moments are independent of /. In the 
model given by (1) and (2), the a’s, \’s, ¢’s, W’s and 6’s 
are all known constants. The finite populations variances 
and covariances of the sensitive questions, 07, op and o,, 
are all unknown. 

For the unrelated questions model (randomized response 
model (i)) assume that the randomizing schemes on the two 
sensitive questions are independent and that sensitive 
question /, 7 = 1, 2, is asked with probability p; and the 
associated nonsensitive questions with probability 1 — p;. 
Assume further that the sensitive questions are unrelated 
to the nonsensitive questions so that o,,, = oy, = Oy = 
0), = 0. This assumption is unnecessary under simple 
random sampling with replacement. When, in addition, 
a random permutation model is assumed on the quadruple 
(xj, ¥j, uj, vj) then in the model given by (1) and (2): 


Oy (1 — p1)U, B, = [Rip Cp = (1 — p2)V, By 25 


¢, = Pi, %o1 = (1 — pion + pi (1 — p,) 0’, 
Yi = —2p,(1 — pi) U, Yo, = pi(1 — py), 


$2 = Pro, Wo = (1 — po)op + po(1 — po) V’, 


Yio = —2p.(1 — po) V, yx = pr(1 — pr), 
é. 2 ie pm) 
Ofer Di Nal); N= el py) “og (N=) 
5. = —p3/(N — 1), 2 = —(1 = py)705/(N — 1), 
$3 = PiP2, 63 = —$3/(N — 1), 


va = (lp) (1 3) 6, 
and 3 = —¥3/(N-— 1). (3) 


Note that the model assumptions require that the finite 
population variance-covariance matrix of the nonsensitive 
questions is known as well as the finite population means. 

For the additive constants model (randomized response 
model (ii)) assume that the random variables wu and vy that 
are added to the value of the responses to the two sensitive 
questions are independent with means yp, and pw, and 
variances oe and o? respectively. When the random per- 
mutation model is assumed on the pair (x;, y;) then in the 
model given by (1) and (2): 


ay = py By = 1, a2. = wy, B. = Ie 
2 
= 04, Yo = 9 


$; = 2 = $3 = 1, Yo 


6; = 6) = 6; = —1/(N — 1), 
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In the multiplicative constants model, two independent 
random variables, uw and v with means yp, and p, and 
variances o? and o? respectively, are multiplied respectively 
by the value of the response on the x-variable and the 
y-variable. When the random permutation model is 
assumed on the pair (x;, y;) then in the model given by 
(1) and (2): 


a, = a, = 0, 8B, = py, Bo. = wy, 
22 2 2 2 

%; = By + oy, d2 = wy + OF, 03 = by Hy, 
2 Ps 

Yo1 = 04, Y2 = oO, 


5, = —p2/(N — 1), 6 = —pe/(N — 1), 
63 = —pypy/(N — 1), and 


Yo. = Wu = Yoo = viz = ¥3 = Ny = AQ = AZ = Oz 
(5) 


4. ESTIMATION OF VARIANCE 
AND COVARIANCE 


Consider estimation of o; so that the appropriate data 
are z; for units / € s. The general class of quadratic esti- 
mators of o; is of the form: 


Cbs = loys ate eB by Rj GF Se bs; z ae ‘S ye Dsij ZiXj> (6) 


JES Jés iAj€eS 


where the coefficients of the z’s are defined for all s, all 
J € sand all pairs (i, /) € s. 

In the context of randomized response, an estimator e, 
in the class defined by (6) is design-unbiased for o; if 
E,E(é,) = o; and is pm-unbiased if E(e,) = o>. Con- 
ditions under which an estimator e, is pm-unbiased are 
obtained upon taking the expectation E of (6) under (1) 
and (2). On equating coefficients in Y°, Y', Y* and o; 
four equations in four unknowns are obtained. The solution 
to these four equations yields the following conditions 
under which estimators in the class defined by (6) are 
pm-biased for oF: 


B3 
E by) = =A,, (7 
>( Lu i) eh (dSied0. tae bola aaa 


JES 


BS + vo 
fi Doi ee = 
(d Lu ' B3 (2 — 52) — doo 


iAj€s 


—(A, + Bo), (8) 


(2arWr — BoWi2) 
E bg. = S Cc > (9) 
a( oy :) Be (d= 8) eee 


JES 
and 


2 (B3 + 22) — (A322 — a2Bo~12 + B30) 
B3 ($2 — 52) — dvr 


= D,. (10) 


E,(bs..) = 


In order to obtain the optimal estimator we need to define 
an associated class of quadratic estimators of 0. This is 
given by 


Crp Sy Coe peck: Ne Csjn.2j + yy Csij zp 45 Me y Cs ij Zi %; « 


JES JES iAJ€s 


The conditions for an estimator e, in this class to be 
pm-biased for 0 are 


E,(Cs..) = E,( Dy cy) = E,( y cu) a 


J€s JES 


etd, aa) = 0. (i) 


iAJEs 


Derivation of the minimum variance quadratic design- 
unbiased estimator of o, follows along the same lines as 
that used for the finite population mean by Rao and 
Bellhouse (1978) for cases without randomized response 
and by Bellhouse (1980) for cases with randomized 
response. The covariance E'(e,e,) under the composite 
expectation is determined under the model such that only 
expectations of the form E, remain to be determined. 
From this expression the coefficients b are set to make 
E(eye.) = Ounder the conditions in (11). The values of 
the coefficients b are then determined from the conditions 
in (7) through (10). From a theorem on minimum variance 
unbiased estimation of Rao (1952), the resulting estimator 
is the optimal pm-unbiased estimator of apse If there exists 
a design such that this estimator is also design-unbiased for 
On then by arguments similar to those given in Theorem 
(2.4) of Rao and Bellhouse (1978), the estimator is also the 
optimal design-unbiased estimator of On We present 
results for pm-unbiased estimators first (Theorems | and 2) 
and then present results for design-unbiased estimators 
under the three randomized response schemes. 
Theorem 1. Under the model defined by (1) and (2) and 
for any design of fixed size n, the pm-variance of e,, 
E,plEpE-(€y — o5)7] = E(ey — o7)*, is minimized for 
the estimator given by 


1 
(Ag +) Bz)s7 = "By zp Cy Zi Dy) 
n 


JES 
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where Z is the sample mean of the data and 


1 
2 =\2 
‘a feat 
Zz 1 (zy Z) 


JES 


is the sample variance of the data obtained through ran- 
domized response where A>, B,, C, and D, are defined in 
(7) through (10) respectively. 


Proof. Under the model given by (1) and (2) the covariance 
E(e,e,) is algebraically quite lengthy but may be expressed 
in the following form: 


b’Gce + H, (13) 


where b? is the vector 


| EC) E,( ms by.) Eo( y bai). 


JES J€s 


E, (x a bu) |. (14) 


iAjes 


and c? is the same as (14) with the b’s replaced by c’s. The 
4 x 4matrix Gin (13) contains functions of the first order 
moments of z; and the second order moments of é; in (1). 
The expression H in (13) is a sum of terms of the form 


«cy Dsij Cskl> (15) 


where the summation symbol is up to a quadruple sum, 
where the subscripts of b could be replaced by a dot (.) and 
where « is a function of second through fourth order 
moments of é; in (1). Note that these moments are all 
independent of /. In (15) the sum is a single sum over / € s 
when, for example, the subscriptsi = 7 = k = /or when 
i = kand/jand /are replaced by dots. The sum is a double 
sum over? #~ k € swhen, forexample,/ # kand/and / 
are replaced by dots. This process continues to the qua- 
druple sum in whichi # j # k # J. From (11) E(e,e,) 
reduces to Oif b, = Ay, gj = ho, Dg = hy, and bei; = ha, 
where the 4; are constants. From (7) through (10) and the 
fact that the design is of fixed size we obtain 


A, + B, 


bs. = Dg, by. = C/N, bs = ie ee 51" 


bs = A2/n, 


so that the estimator in (12) minimizes the variance in the 
pm-unbiased class of quadratic estimators of 07. Q.E.D. 


By the same arguments 


1 
GAiedIB SS ye kBpesn Ya! w7i ERC {WIELD 5 wuwi(L6) 
n 


JES 


WT 


is the optimal pm-unbiased estimator for o where 


2 
A, = 2 Bi ’ 
Bi(d; — 6)) — dio 
pie Bi + vo 
ty in Dy ’ 
Br (od, — 6;) — dv 
Cxe Pena — Bov1) Peat 
Br (od, — 6)) — diay 
Des Mi (Bi + Yo) — (ata — aBidn + Bio) 


Bi (d, — 5;) — 8; Y21 


The same technique can be used to estimate the co- 
variance o,,. The general class of quadratic estimators of 
0xy is of the form 


Cgs = ds + D> Gi jap 3 dy; Wj ayo ys dij Wi Zj5 


J€s Jj€s ixj€s 


where the coefficients of the w’s and z’s are defined for 
all s, all j € s and all pairs (i, /) € s. The result on the 
covariance is stated without proof in 


Theorem 2. Under the model defined by (1) and (2) 
and for any design of fixed size n, the pm-variance of 
Cg, EplEpE,(€¢ — Oy)7] = E(eq — oy)’, is minimized 
for the estimator given by 


Swz (¥3 = Aa) (17) 
$3 — 63 
where 
1 = a 
Aiea al 1 » (wr) Cz; S12) 


is the sample covariance between w and z. 


Anestimator for p is obtained from (12), (16) and (17). 
In the additive constants randomized response model 
(randomized response model (ii)) the estimator of p is 
given by 


A Swz 
pa (18) 
(sz, — 02) (s2 — 0?) 


This is the same as the estimator obtained by Edgell et al. 
(1986). Under the multiplicative constants model (random- 
ized response model (iii)) the estimator reduces to 
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Pme = 
Swz 
0. 1 o2/ 1 
32 by fie oe w? s2 Be Er, 
ir Ou! ba N jes Lit oy / by nN jes 


(19) 


for un, ~ Oandp, # 0. When p, = 0 the coefficient of 
yw? is 1/n and when p, # 0 the coefficient of Y z} is 
1/n. The estimator for p under the unrelated questions 
model (randomized response model (1)) is 


Chapel — Pa) ¢ 
WZ uv 
: P\P2 
COS ae eras rare ae a See a (20) 


crs 


where S,, = No,,/(N — 1) and where 


- 1 o 
Satu = Pi) Wa ls Cn 


JES 


a 1—p 
(lps 1G ~ pyai(r ira 4) 
and 


p 1 £ 
Stisuse aid ~Pad= ia + 20 — pr) Vz — 
JES 


(l-p)V- (1 ~ pret ze =m) ; 


Naat 


When p; = p> this may be compared to the estimator in 
Edgell et a/. (1986). The resulting estimator for 6,, differs 
from the estimator in Edgell et a/. (1986) who assume that 
0, = 0. They also use biased estimators of 02 and pe 
Edgell et a/.’s estimator for o, is obtained by writing the 
design variance of Z under simple random sampling with 
replacement as 


N 
G7 i= yia( Zp Za) (NID) (21) 


j=l 
The design variance of Z under the randomizing device is 
[proy + (1 — pr)oy + po(1 — pn) (Y—V)?]/n. (22) 


Expression (22) is found in Greenberg ef al. (1971). The 
estimator for o; is found by equating (22) to the left hand 
side of (21), by substituting sample the estimator of o? 
and the randomized response estimator of Y in the 
resulting equation, and then by solving for o,. 


Each of the estimators of the finite population variances 
and covariance, which are the components of @ in (18), (19) 
and (20), are design-unbiased under the appropriate ran- 
domized response model for any design with joint inclusion 
probability for units 7 and / given by mj = n(n — 1)/ 
[N(N — 1)]. Consequently, each estimator is the optimal 
design-unbiased estimator for its finite population para- 
mater counterpart. To obtain the appropriate unbiased 
estimators in (18), multiply the numerator and denomi- 
nator each by (N — 1)/N. The resulting numerator is 
design-unbiased for o,, and the expressions under the 
square root Sigil in the denominator of (18) are unbiased 
for o and One In (19) it is necessary to multiply the 
numerator and denominator by (N — 1)/[Ny,p,] in 
order to obtain the correct form of the design-unbiased 
estimators. The correct estimators are obtained in (20) 
when the multiplier is (NV — 1)/(Np,p2). 

In any of the randomized response designs, the simplest 
estimate of the variance of 6 is the jackknife estimate of 
variance. Jackknife estimates of variance for 6 can be 
obtained from formulae (4.2.3) or (4.2.5) in Wolter (1985). 


5. EFFECT OF RESPONSE BIAS 


In the additive constants model, the respondent is asked 
to add a random variable u to x and an independent 
random variable v to y. Instead, the respondent may add 
different independent random variables, say u’ and v’. 
The means and variances of uw’ and v’ may differ from 
those of wand v. It is reasonable to assume, however, that 
a2, = of and o2, = o2. One example in which this situa- 
tion might occur is the following. The respondent does not 
want to add on the outcome of a random variable near to 
the mean of the distribution of the random variable. In 
this case the distribution of response bias could be modelled 
by the original distribution with an interval around the 
mean In which any outcome from the original distribution 
which falls in the chosen interval is set to one of the end 
points of the interval. On taking separately the expectations 
of the numerator and the expression under each of the 
square root signs in the denominator of (18) the expression 


to 
2 2 “7 oh (23) 
Jo2 sur Unis = o2 Io? FP GH, = OR: 


is obtained. From (23) it may be noted that the response 
bias leads to an estimate of correlation lower than the true 
value. 

The multiplicative constants model is the same as the 
additive constants model with the exception that the 
responses to the sensitive questions are multiplied by the 
random variables. As in the response bias model for 
additive constants, assume that uw’ and v’ are used by the 
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respondent instead of uw and v. Then on taking separately 
the expectations of the numerator and the expressions 
under each of the square root signs in the denominator of 
(19) the expression 


: : =, = 2 2 2 2 
is obtained. If uw, = wy, wy = My, O14 = o,and oy, =o, 


as in the case of the additive constants model, then from 
(24) the response bias leads to an overestimate of the 
correlation. 

In the unrelated questions model a reasonable model 
for response bias is to assume that the sensitive questions 
are answered with probability pj < p,; and pz < p>. In 
general the effect of this response bias is dependent on the 
relative values of the various probabilities, the means and 
variances of the sensitive questions, and the means and 
variances of the nonsensitive questions. Under simple 
random sampling without replacement and the response 
bias model, the design expectation of the numerator of (20) 
is given by 


rae E eer ee rs 1) PD) Su 


PiP2 


which is greater than p; p3S,,. Likewise the design expec- 
tation of $2 in (20) is 


S2[pi? + (N — 1)p{ (p, — pi)/N] 
+ (p, — p{)Sz[pi — (pi + 2p, — 2)/N] 


+ pi(p, — pi)(X — 0)’, 


which is greater than pj*S2 when Nis large. If S,,, = 0, 
then the response bias leads to an underestimate of the 
correlation. 
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On Efficiency of Using Distinct Respondents 
in a Randomized Response Survey 


N.S. MANGAT, R. SINGH, S. SINGH, D.R. BELLHOUSE and H.B. KASHANI! 


ABSTRACT 


It is well known that the sample mean based on the distinct sample units in simple random sampling with replacement 
is more efficient than the sample mean based on all units selected including repetitions (Murthy 1967, pp. 65-66). 
Seth and Rao (1964) showed that the mean of the distinct units is less efficient than the sample mean in sampling 
without replacement under the same average sampling cost. Under Warner’s (1965) method of randomized response 
we compare simple random sampling without replacement and sampling with replacement when only the distinct 


number of units in the sample are considered. 


KEY WORDS: Simple random sampling with and without replacement; Inferences with distinct units; Warner’s 


technique. 


1. INTRODUCTION 


The randomized response (RR) technique to procure 
trustworthy data for estimating the proportion of the 
population belonging to a sensitive group was first intro- 
duced by Warner (1965). Since then many developments 
have taken place in this area. Recently, among others, 
Franklin (1989), Kuk (1990), Mangat and Singh (1990, 
1991), Mangat, Singh and Singh (1992) and Mangat (1994) 
have suggested alternative RR procedures/estimators. 

In the usual simple random sampling (SRS) with 
replacement (WR) surveys, it is well known that the esti- 
mator of population mean based on the distinct units is 
always more efficient than the mean based on all selections 
(Murthy 1967, pp. 65-66). Also, Seth and Rao (1964) 
showed that, under the same average cost to sample, 
sampling without replacement was more efficient than 
with replacement sampling using the mean of the distinct 
sample units. This motivated the authors to investigate 
whether the above observation also holds in the case of 
Warner’s pioneer RR model which is widely used in practice 
for selecting the respondents in the case of a survey dealing 
with. sensitive characters. To investigate the problem we 
shall consider the use of four sampling strategies. 


1.1 Strategy I 


According to this (Warner’s) procedure, each respondent 
included in the sample using the SRSWR method is pro- 
vided with a suitable randomization device consisting of 
two statements of the form: (i) ‘‘I belong to sensitive 
group’’ and (ii) ‘‘I do not belong to sensitive group’’, 
represented with probabilities p and (1 — p), respectively. 
The respondent answers ‘‘yes’’ or ‘‘no’’ according to the 


randomly selected statement and to his actual status with 
respect to the attribute, without revealing the statement 
chosen. If m’ persons in the sample (including repetitions) 
answered ‘‘yes’’, Warner’s estimator 


n’/n—1 + 
a = FD ud (1) 
2p—1 


is unbiased for zm and its variance is given by 


“(1 *) pUle=-p) 
n n(2p — Gere 


ViGa)-= (2) 


The value of p should be chosen as close to 1 or 0 as 
possible without threatening the degree of co-operation by 
respondents. 


1.2 Strategy Il 


A sample of n respondents is drawn from a finite popu- 
lation of N units using SRSWR but the information from 
the d distinct units inthe sample, 1 < d < n, is used inthe 
construction of the estimator. Let d’ denote the respondents 
reporting a ‘‘yes’’ answer in the interview conducted with the 
RR device. We then consider the following estimator for 7: 


d’/d—1 
Fo Sapa LP i es, (3) 
2p—-—1 


Conditional on d distinct units, the resulting sample is 
a simple random sample without replacement of size d 
from N units. The estimator 7, is, therefore, unbiased for 
the population 7. 


' N.S. Mangat, R. Singh and S. Singh, Punjab Agricultural University, Ludhiana-141004 (India); D.R. Bellhouse, University of Western Ontario, 
London, Ontario, Canada, N6A 5B7; H.B. Kashani, West Oregon State College, Monmouth, OR 97361, U.S.A. 
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In order to study the performance of the proposed 
estimator 74, we need its variance. We give here the 
expression for the conditional variance V,(7,) for a given 
value of d. Thus 


N= Fa C= 2708) 
| d 


PUS—sp) 


Vita) = setae es 


(4) 


If E, and V, are the expectation and variance over all values 
of d, then we have Vij(ag) = FE, V2(ag) + ViE (iQ). 
On using (4) one gets 


1 1 — 
Vii(%q) = Ne 6 = 1 ae 


DUe="p) . 
‘opengis ({) 2 


since the second term in Vj;(7g) is zero as E(atg) = 7. 


1.3 Strategy III 


The sample of n respondents is selected using SRSWOR 
(Kim and Flueck 1978). In this case the variance of the 
estimator 7 in (1) can be written by replacing d in (4) by 
n. Thus we have 


N-—n x(1 — zm) 
N- 1 n 


Bp) 


Vin (7) = 4Op rr 12" 


(6) 


1.4 Strategy IV 


Here the estimator is based on a WOR simple random 
sample of size E(d). This yields the same expected cost 
for both in SRSWR and SRSWOR. For this scheme the 
estimator will be 


d’/E(d) —1 + 
2p — 1 


with variance 


NIE) 1 


Viv (te) = ae 


m(1 — zm) 


p(1 — p) 
E(d)(2p — 1)? 


(7) 


2. EFFICIENCY COMPARISONS 


It has been shown by Korwar and Serfling (1970) that, 
imei = Be 


1 1 


where 


Let us now examine the variance expression in (5). 
Using Q, it is easily verified that 


NE,(1/d) — 1 _ ! 8) 
Ne=aid arriba 


in the first term on the right of (5) but that E;(1/d) = 1/n 
in the second term on the right of (5). Thus the relative effi- 
ciency of the SRSWR estimator in (1) using repeated units 
with respect to the SRSWR estimator in (3) using the 
distinct number of units will depend on the relative sizes 
of x and p. This is due to the fact that the repeated units 
can give rise to different responses because of the ran- 
domizing device and hence can provide some additional 
information. A sufficient condition for the inequality 
Vii (tg) — Vi(iw) < 0 to hold is obtained by using 
E,(d) = Q. Thus we get the condition as 


n(N—1)(6N+n—1) p(1—p) 
Ni6Nn— 12N— nr 1) Op 
(9) 


a(1—z) > 


The above inequality is likely to hold for values of p closer 
to 0 or 1, the situations in which respondent jeopardy would 
be of concern. For example, if N = 100, n = 10 and 
p = 0.9, the inequality (9) will hold for 0.236 < a < 0.764. 

Similarly, Strategy II will be inferior to Strategy I if 
Vit(tg) — V(r) > 0. Using E,(1/d) = O — 1/720N 
this inequality reduces to 


a(1— 7) 


n(N — 1){359N + 60(n —1)} p(1 —p) 
N{361Nn — 720N — 60n(n — 1)} (2p — 1)?” 


This inequality will hold for the example considered for 
inequality (9) whenever either t < 0.234 or t = 0.764. 


On using the Cauchy-Schwarz inequality, E(1/d) > 
1/E(d), as in Seth and Rao (1964) we find that Vj(7z) > 
Viv (ig). This implies that Strategy IV is more efficient 
than Strategy II. 

It is trivial to note that Strategy III is more efficient than 
Strategy I. 
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We know that E(1/d) = 1/n. This means Yi; (ag) > 
Vi; (7), implying that Strategy III is more efficient than 
Strategy II. 

Since 1/E(d) = 1/n, Strategy II is more efficient than 
Strategy IV. 

The last pair to consider consists of Strategies I and IV. 
Since E(1/d) > 1/E(d) forn > 1, on using (8) we have 


NYE CA) ak : 1 


N- 1 n 
implying that in (7) and (2) 
N — E(d) x(1 — om) J a(1 — mr) 
N-—- 1 i () n 


for n > 1. Also 1/E(d) = 1/n. This shows that the 
second term of (7) on the right hand side will be more than 
the corresponding term of (2). Thus the relative efficiencies 
of Strategies I and IV depend on relative values of 7 and 
p. As a numerical illustration, if N = 100, nm = 10 and 
p = 0.9 then Strategy IV will be more efficient than 
Strategy I for 0.18 < a < 0.82. 
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Cross-sectional Weighting of Longitudinal Surveys of Individuals 
and Households Using the Weight Share Method 


PIERRE LAVALLEE! 


ABSTRACT 


Statistical agencies are conducting increasing numbers of longitudinal surveys. Although the main output of these 
surveys consists of longitudinal data, most of them are also expected to produce reliable cross-sectional estimates. 
In surveys of individuals and households, population dynamics significantly changes household composition over 
time. For this reason, methods of cross-sectional estimation must be adapted to the longitudinal aspect of the sample. 
This paper discusses in a general context the Weight Share method, of which one application is to assign a basic 
weight to each individual in a household. The variance estimator associated with the Weight Share method is also 
presented. The weighting of a longitudinal sample is then discussed when a supplementary sample is selected to 
improve the cross-sectional representativeness of the sample. The paper presents as an application the Survey of 
Labour and Income Dynamics (SLID) introduced by Statistics Canada in 1994. This longitudinal survey covers 
individuals’ work experience, changes in income and changes in family composition. 


KEY WORDS: Weight share method; Longitudinal survey; Cross-sectional estimate; Supplementary sample. 


1. INTRODUCTION 


Longitudinal surveys, /.e. surveys that follow units over 
time, are steadily gaining importance within statistical 
agencies. Statistics Canada is currently developing three 
major longitudinal surveys of individuals: the National 
Population Health Survey, the National Longitudinal 
Survey of Children; and the Survey of Labour and Income 
Dynamics (SLID). 

The primary objective of these surveys is to obtain longi- 
tudinal data. One of the uses of these data is to study the 
changes in variables over time (e.g., longitudinal data may 
be used to analyze the chronic aspect of poverty). A secon- 
dary objective is the production of cross-sectional estimates, 
in other words estimates that represent the population at 
a given point in time. Although these estimates are far less 
important than the longitudinal data, to many users they 
are an essential aspect of the survey. Obtaining a represen- 
tative cross-sectional view of the current population 
constitutes a means of measuring changing situations over 
time. The longitudinal aspect of the survey also improves 
the accuracy of the measurement of change. 

This paper presents an extension of the Weight Share 
method presented by Ernst (1989). Although the method 
has been developed in the context of longitudinal household 
surveys, it is shown that the Weight Share method can be 
generalized to situations where a population of interest is 
sampled through the use of a frame which refers to a 
different population, but linked somehow to the first one. 
In the context of longitudinal surveys, the frame can be 
associated to the initial population, while the population 
of interest can be the population a few years later. The 


paper also provides a new proof of the unbiasedness of the 
Weight Share method together with the variance formula 
and variance estimator to be used with the method. 

Using the Weight Share method, the question addressed 
in this paper is that of ensuring that the longitudinal 
sample can be used for cross-sectional estimation. The 
difficulty arises from the fact that, although the longitu- 
dinal sample remains constant, distribution of the popula- 
tion (individuals and households) changes over time. At 
the individual level, these changes are produced by such 
events as births and deaths, immigration and emigration, 
and moves within the country. Obviously, the birth or death 
of an individual also changes household composition; and 
such events as marriage, divorce, separation, departure of 
a child and cohabitation, are all factors that affect popula- 
tion distribution within the household. If we are to obtain 
accurate, unbiased cross-sectional estimates based on a 
longitudinal sample, we need an estimation method that 
takes these changes into account. 

Our initial topic is the presentation of the Weight Share 
method in a general context. Secondly, we present the sample 
design for SLID. This is one of the major longitudinal surveys 
for which the production of cross-sectional estimates from 
a longitudinal sample is a significant problem. The survey 
itself is a typical longitudinal survey of individuals and 
households. Thirdly, we describe the use of a supplemen- 
tary sample added to the initial longitudinal sample to 
improve the cross-sectional representativeness. Fourthly, 
we present the concept of basic weights, the equivalent, 
as it were, of sample weights. Finally, we describe the use 
of the Weight Share method to calculate basic weights for 
all individuals interviewed in SLID. 


! Pierre Lavallée, Social Survey Methods Division, Statistics Canada, Ottawa, Ontario, Canada, K1A OT6. 
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2. THE WEIGHT SHARE 
METHOD IN A 
GENERAL CONTEXT 


The Weight Share method is described in Ernst (1989) 
in the context of longitudinal household surveys. In the 
same context, Kalton and Brick (1995) discuss different 
weighting schemes, including the Weight Share method. 
Various implications of using the Weight Share method 
for longitudinal household surveys have been described by 
Gailly and Lavallée (1993). 

We now present this method in a general context that 
can be applied to several sampling situations where the 
population of interest needs to be sampled through the use 
of a frame which refers to a different population, but is 
linked somehow to the first one. Note that this can be 
viewed as a form of Network Sampling (see Thompson 
1992). For example, one can imagine the need to sample 
young children where the only available frame is a list of 
names of parents. The population of interest is really the 
children but we need to select a sample of parents from 
the frame in order to obtain the sample of children. Note 
that the children of a particular family can be sampled 
through either the father or the mother. Another example 
is one of business surveys where an incomplete frame of 
establishments is available. For each selected establishment 
from the frame, we wish to sample the entire set of estab- 
lishments belonging to the same enterprise. The missing 
establishments from the frame are expected to be sampled 
via the establishments present on the frame. 

Suppose that a sample s@ of m4 units is selected from 
a population U4 of M% units using some sampling 
design. Let af be the selection probability of unit 7. We 
assume 1/1 > 0 for all j € U4. 

Let U? be a population of M? units. This population 
is divided into N clusters where cluster i contains MP 
units. For example, in the context of social surveys, the 
clusters can be households and the units can be the persons 
within the households. For business surveys, the clusters 
can be enterprises and the units can be the establishments 
within the enterprises. From population U®, we are inter- 
ested in estimating the total Y= Y™, YM, yy for 
some characteristic y. 

An important constraint that is imposed in the measure- 
ment (or interviewing) process is to consider all units 
within the same cluster. That is, if a unit is selected in the 
sample, then every unit of the cluster containing the selected 
unit will be interviewed. This constraint is one which often 
arises in surveys for two reasons: cost reductions and the 
need for producing estimates on clusters. Referring back 
to the example of social surveys, there is normally a small 
marginal cost for interviewing all persons within the 
household. On the other hand, household estimates are 
often of interest with respect to poverty measures, for 
example. 


We assume that there exists a /ink (or a correspondence) 
between each unit j of population U4 and at least one 
unit k of population U®. Also, each cluster i of U? has at 
least one link with a unit j of U4. The link is identified 
through an indicator variable /;, where /;, = 1 if there is 
a link between unit j € U4 and unit k € U? and 0 other- 
wise. All units of population U4 have at least one link 
with population U®, i.e., LA = Yxevs lx = 1 for all 
j € U4. However, there can be zero, one or more links 
for a unit k of population U®, i.e., it is possible to have 
Ce ye — 0 OL Le) eu tor some 
k € U®. This is illustrated in Figure 1. 


i ; 


Figure 1. Links between units of populations U4 and U®. 


The estimation process presented now uses the sample 
s4 together with the links existing between U4 and U? to 
obtain an estimation of the total Y belonging to population 
U®. The links are in fact utilized as a bridge to go from 
population U% to population U®, and vice versa. Note 
that in practice, it might not be physically possible to 
directly select a sample s? from U®, as it has been described 
in the introductory examples. 


To estimate the total Y, one can use the estimator 


nut 
i= ay Nes Wik Vik» (1) 
k=1 


i=1 


where v is the number of interviewed clusters and wi, 
is the weight attached to unit & of cluster 7. To obtain 
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unbiased estimates, a possible set of weights could be the 
inverse of the selection probabilities of the units entering 
into the estimator Y. For each unit & of cluster 7 having 
a link /;;, = 1 with a unit / in U4, this is possible since 
we have 7? = 14. However, not all units of U? neces- 
sarily have a link to U*. Moreover, even if a link exists, 
it is not guaranteed that the selection probability aA is 
known when j ¢ s4; the sample design used to select s4 
could be, for example, a multistage sample design where 
the ultimate selection probability of each unit / is only 
known at the end of the selection process. To assign a 
nonzero weight w;, to each unit k of cluster / entering into 
Y, the Weight Share method can be used. 

In general, the Weight Share method allocates to each 
sampled unit a basic weight established from an average 
of weights calculated within each cluster i entering into Y. 
An initial weight that corresponds to the inverse of the 
selection probability is first obtained for unit k of cluster 
iof Y having a link /;;, = 1 witha unit / €s%. An initial 
weight of zero is assigned to units not having a link. The 
basic weight is obtained by calculating the mean of the 
initial weights for the cluster. This weight is finally assigned 
to all units within the cluster. Note that the fact of allo- 
cating the same basic weight to all units has the consid- 
erable advantage of ensuring consistency of estimates for 
units and clusters. 

Formally, each unit & of cluster i entering into Y is 
assigned an initial weight w;, as follows: 


A 
por a fj 

Wik = y Lik > (2) 
= Tj 


where ¢; = lify € s4 and 0 otherwise. Note that a unit k 
having no link with any unit j of U4 automatically has an 
initial weight of zero. 


The basic weight w; is given by 


M, 
Ny Wik 
se igice atf (3) 


ve 
Ly 
k=I 


where L;, = ri I; ix. The quantity Lj, represents the 
number of links between the units of U4 and the unit k 
of cluster i of population U?. The quantity L; = ¥ pal Lh 
then corresponds to the total number of links present in 
cluster /. 


Wi 


Finally, we assign w;, = w,; for all k € 1. 


2.1 Unbiasedness of the Weight Share Method 


We now show that the estimator Y with the Weight 
Share method is unbiased for Y. Starting with Y = 


aif 


yy MG Dik = L 7=1 wiY;, we replace the definition of 
w; in Y to get 


MP 
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Letting z;, = Y;/L; for all k € i, we then have 


n Mp 
Tye Wye wikzix: (4) 
i=1 k=l 
Let a single index k be used to identify the m? units entering 
into Y(m® = y 7_, M®). By replacing w; by its definition 
(2), we obtain 
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Now since ¢; # 0 only for the units & entering into Y, we 
can extend the first summation to all units k in U?. That is, 
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Now, taking the expectation gives 
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since E(t;)) = af. 
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It suffices now to show that Z = Y. First, we have 


M4 oP 
2-5) 4-0 ise Da Ele 
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By rearranging these summations in terms of the N clusters 
of population U?, we then obtain 


A 


N 
z=y Ya 
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=) 7 be = = Y. 
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The unbiasedness of the Weight Share method can also be 
proved using an approach similar to the one presented by 
Ernst (1989). 


2.2 Variance Estimation 


To obtain a variance formula for Y, we start with 
equation (5). Since Y turns out to be nothing more than 
a Horvitz-Thompson estimator of Z (see Horvitz and 
Thompson 1952), the variance of Yis then directly given by 


Var(Y) = 


where wh, is the joint probability of selecting units / and 
Jj’ (see Sarndal, Swensson and Wretman 1992 for the 
calculation of TH, under various sampling designs). 

In practice, equation (6) is simple to use. Initially, it 
suffices to calculate hu Y;/L; for each unit k € 7. Then, 
we compute Z; = => pel li Z,- All that remains is to plug 
each Z; into the variance equation of the Horvitz- 
Thormpeon estimator. 

The variance Var(Y) may be unbiasedly estimated 
from the following equation: 


A A 
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Another unbiased estimator of the variance Var ( Y) may 
be developed in the form of Yates and Grundy (1953). 
Other variance estimators are available in the literature, 
such as jackknife variance estimators. A jackknife variance 
estimator in the context of the SLID sample design is 
discussed in Section 3.2.3. For further details, see Wolter 
(1985) and Sarndal, Swensson and Wretman (1992). 


3. APPLICATION TO SLID 


In January 1994, SLID was launched by Statistics 
Canada. Its aim is to observe individual activity in the 
labour market over time, and changes in individual income 
and family circumstances. To repeat, the primary aim of 
SLID is to provide longitudinal data. However, cross- 
sectional estimates will also be produced. The target 
population of SLID is all persons, with no distinction as 
to age, who live in the provinces of Canada. For opera- 
tional reasons, the Territories, institutions, Indian reserves 
and military camps are excluded (see Lavallée 1993). 


3.1 Sample Design 
3.1.1 Initial Sample 


The SLID longitudinal sample was drawn in January 
1993 from two groups rotating out of the Canadian Labour 
Force Survey (LFS), making the sample a sub-sample of 
the LFS. The longitudinal sample for SLID is made up of 
close to 15,000 households. A household is defined as any 
person or group of persons living in a dwelling. It may 
consist of one person living alone, a group of people who 
are not related but who share the same dwelling, or it may 
be a family. 

LFS is a continuing survey designed to produce monthly 
estimates of employment, self-employment and unem- 
ployment. This survey uses a stratified multi-stage design 
which uses an area frame in which dwellings are the final 
sampling units. All the individuals who are members of 
households that occupy the selected dwellings make up the 
LFS sample. In other words, LFS draws a sample of 
dwellings and all individuals in the households that live in 
the selected dwellings are interviewed. A six-group rotation 
plan is used to construct the sample: every month, one 
group that has been in the sample for six months is rotated 
out. Each rotation group contains approximately 10,000 
households, or approximately 20,000 individuals 16 years 
old or more. For further details on the LFS sample plan, 
see Singh ef al. (1990). 

For SLID, the longitudinal sample will not be updated 
following its selection in January 1993. However, to give 
the sample some cross-sectional representativeness, initially- 
absent individuals in the population (i.e. , individuals who 
were not part of the population in the year the longitudinal 
sample was selected) will need to be considered in the 
sample in January 1994 and later. Initially-absent individ- 
uals include newborns (births since January 1993) and in- 
migrants. Note that this addition to the sample will be 
cross-sectional in that only the longitudinal individuals will 
be permanently included in the sample. 

Table | presents the terminology developed for SLID. 
After sample selection in January 93 (year 1), the popula- 
tion contains longitudinal individuals and initially-present 
individuals. In January 94 (year 2), the population contains 
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longitudinal individuals, initially-present individuals and 
initially-absent individuals. Focusing on the households 
containing at least one longitudinal individual (i.e., 
longitudinal households), initially-present and initially- 
absent individuals who join these households are referred 
to as cohabitants. 


Table 1 
SLID Terminology 


Individuals: 


Longitudinal individuals: Individuals selected at year 1 in the 
longitudinal sample. 


Initially-absent individuals: Individuals who were not part 
of the population in the year the longitudinal sample was 
selected (year 1). It includes in-migrants and newborns. 


Initially-present individuals: Individuals who were part of the 
population of year 1 but were not selected then. 


Cohabitants: Initially-absent and initially-present individuals 


who join a longitudinal household. 


In-migrants: Individuals who, in January of year 1, were 
outside the ten provinces of Canada and individuals in 
excluded areas (the Territories, institutions, Indian reserves 
and military barracks). 


Newborns: Births since January of year 1. 
Households: 


Longitudinal households: Households containing at least one 
longitudinal individual. 


SLID will follow individual and household charac- 
teristics over time. At the time of each wave of interviews, 
all the members of a longitudinal household will be inter- 
viewed. The composition of the longitudinal households 
will change over time, as the result of a birth or the arrival 
of an in-migrant in the household. A part of the selection 
of initially-absent individuals may be based on individuals 
who join longitudinal households. 


3.1.2 Supplementary Sample 


The restriction to initially-absent individuals who join 
longitudinal households will unfortunately exclude house- 
holds made up of initially-absent individuals only (e.g., 
in-migrant families). To offset this shortcoming, one pos- 
sibility is to draw a Supplementary Sample. This sample 
could be one of dwellings drawn directly from the ongoing 
LFS at each wave of interviews. Supplementary questions 
would then be added to the LFS questionnaire to detect 
households that contain at least one in-migrant; the house- 
holds selected would then be interviewed. 

Recalling that the Supplementary Sample is used for the 
selection of households made up solely of initially-absent 
individuals (i.e. , in-migrants and newborns), restricting this 
sample to in-migrants only would not cause any represen- 
tativeness problem. This is because it is highly unlikely that 
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households containing only newborns would be found: 
each household normally contains at least one adult. The 
newborns are then already represented in the sample by 
the longitudinal households. Now, if the Supplementary 
Sample were to include newborns in addition to in-migrants, 
significant costs would be added to the survey. This is because 
the Supplementary Sample would include a complete 
household for each newborn selected in the Supplementary 
Sample, producing excessive sample growth and unnecessary 
costs since the newborns are already represented in the 
sample. 

One other approach different from using the ongoing 
LFS could be to select the Supplementary Sample by 
revisiting the dwellings used for the selection of the initial 
sample. This method offers some practical advantages, one 
being the facility to go to known addresses. This approach 
however would bring the problem of new dwellings which 
were not there in January 1993. These dwellings would 
have a zero probability of being selected in the Supplemen- 
tary Sample and a bias would therefore be introduced. 
This is one reason favouring the first approach, i.e., 
detecting households that contain at least one in-migrant 
via the questionnaire of the ongoing LFS. 


Figure 2. Selection of persons for SLID. 


Figure 2 summarizes the longitudinal and cross-sectional 
selection of individuals. In Figure 2, the letters and houses 
represent individuals and households, respectively. Individ- 
uals A, D, Eand F are longitudinal individuals whom we 
follow over time. Individual C is an initially-present indi- 
vidual, i.e., an individual who was included in the population 
in year | but was not selected then. Initially-absent and 
initially-present individuals who join a longitudinal house- 
hold are called cohabitants. In year 2, individual H represents 
an initially-absent individual who joins the sample as a 


30 Lavallée: Cross-sectional Weighting Using the Weight Share Method 


cohabitant. The fourth house in year 2 represents a house- 
hold selected for the Supplementary Sample of year 2 and 
in which individuals I and J are initially-absent individuals 
(with one of the two being necessarily an in-migrant since 
the Supplementary Sample is restricted to them). Indi- 
vidual G is an initially-present individual with the same 
status as C. In year 3, individuals C and H have left their 
longitudinal households and will therefore not be inter- 
viewed. Individuals I and J who were selected in the Sup- 
plementary Sample are now replaced with the individuals 
of the Supplementary Sample of year 3, i.e., individuals 
K and L. Individual M is an initially-absent individual 
joining a longitudinal household as a cohabitant. It may 
finally be noted that, for cross-sectional purposes, a 
selected household may contain one or more longitudinal 
individuals, initially-present individuals and initially-absent 
individuals (newborns and in-migrants). 


3.2 Basic Weighting 


3.2.1 General Considerations 


To produce cross-sectional estimates, the longitudinal 
sample augmented with initially-absent individuals and 
initially-present individuals must be weighted. The first 
step is to obtain a basic weight for each individual in each 
interviewed household. The basic weight is the weight 
prior to adjustment or post-stratification. It is, so to speak, 
the equivalent of the sample weight. Note that the basic 
weights are useful solely for cross-sectional estimation. 

The basic weights are obtained from the selection prob- 
abilities. As described above, in January 1993 (year 1), we 
select for SLID a sample s“) of m" individuals from a 
population U“) of M“” individuals. The sample is selected 
through dwellings which contain households. In other 
words, the m“') individuals are obtained by selecting n“! 
households from N“!), each household J being selected 
with probability 7{!) > 0,7 =1, ..., N“. Let Mf” be 
the size of household Jso that MW“) = y MOM), Also 
let 7{") be the selection probability of individual j. This 
selection probability is retained throughout all waves of 
the survey. 

For a given subsequent wave (which may be defined as 
year 2), the population U contains the M“!) individuals 
present at year 1, plus some M™) initially-absent individ- 
uals (/.e., initially absent from the population at year 1). 
The population of initially-absent individuals is indicated 
by U). Hence, the population U = U“) U U™ contains 
M = M") + M”) individuals. Letting U*® be the popu- 
lation of M*® in-migrants of year 2, we have U*® ¢ U) 
and also M*® < M). In our notation, the asterisk (*) 
is used to specify that the newborns have been excluded. 
The individuals of year 2 are contained in N households 
where household / is of size M;,i = 1, ..., N. 

For cross-sectional representativeness, some in-migrants 
are selected from the Supplementary Sample. At year 2, 


we then select a sample s*® of m*® individuals from the 
population U*® of M*® in-migrants. The Supplementary 
Sample is selected through households, i.e., the m*? 
individuals are obtained by selecting n*® households. 
Let 1*” be the selection probability of the in-migrant /. 
We assume 1*® > O for j = 1, ..., Mr: 

One implication of selecting in-migrants through house- 
holds is that other individuals (such as newborns, initially- 
present individuals or longitudinal individuals) can be 
brought in by the Supplementary Sample by living in the 
same household as the selected in-migrants. However, 
since the selection units of the Supplementary Sample are 
restricted to the in-migrants, these other individuals are 
not properly selected, say, in the Supplementary Sample, 
even if they will be interviewed. The selection probabilities 
of these individuals are in fact not well defined. 

The remaining in-migrants selected for cross-sectional 
representativeness are those individuals who join longitu- 
dinal households, who are then considered as cohabitants. 
As with the newborns and initially-present individuals of 
the previous paragraph, the addition of cohabitants to 
longitudinal households brings individuals with non-well 
defined selection probabilities. 

The individuals with non-well defined selection proba- 
bilities have entered the survey process in a ‘‘non-legitimate”’ 
way. They complicate the determination of the basic 
weights, as their selection probability is not well defined. 
In order to override this difficulty, the Weight Share 
method is proposed. 


3.2.2 Basic Weight Calculation 


The Weight Share method described in Section 2 is now 
applied to the SLID sample, including the Supplementary 
Sample. The population U4 is here represented by the 
union of the two distinct populations U" and U*™, ie., 
U4 = Ut = U") + Ut, The sample s4 of m = m“) + 
m*®) individuals corresponds to the union of the two 
distinct samples s“!) and s*. The population U? is repre- 
sented by U = U“!) + U®), The population U4 = U* 
excludes the newborns while the population U? = U 
includes them. The clusters of population U® simply corre- 
spond to the N households of year 2, and hence M2 = M,. 

One possible linkage between population U4 and U? 
can be established by the same individuals in populations 
U4 and U®. That is, /;, = 1 if individual jin population 
U4 corresponds to individual k in population U?, and 
li, = 0 otherwise. For each individual k not being a new- 
born, we then have L;, = ee Liz = 1. On the other 
hand, for each newborn k, we have Lj, = Y ja, lin = 0 
since they are excluded from U4. We now have L; = 
YM, Lick = Mt where M# is the size of household i 
excluding the newborns. 

Note that this last linkage is only one among several 
other possibilities. One other possible linkage would be to 
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extend the linkage of the previous paragraph to all other 
persons within the household. That is, assign /;, = 1 for 
all individuals k (of U?) belonging to the same household 
i where individual j (of U4) now belongs in U?, and 0 
otherwise. In other words, /;, = 1 if individuals j and k 
belongs to household 7. For each individual k in household 
i, we then have Liz =, yi ix = M}. We also get 
EL, = YofiLlae = L/, Mt = MPM}. One can show 
that this linkage produces the same basic weighting as the 
one from the previous paragraph. Because the first linkage 
corresponds to a more natural way to link the individuals 
(i.e., by linking only the same individuals between U4 and 
U®), we will adopt the linkage proposed in the previous 
paragraph. 

By considering the definition (2) of the initial weight 
w/, of individual k in household /, we obtain 


(1) tr) 
Wik sean 6)? (8) 


where ¢;\!) = 1 if individual k is part of s“!) and 0 other- 
wise, ¢%?) = 1 if individual k is part of s*® and 0 other- 
wise. This can be written more explicitly as 


tyme tor kes? 
Wiese li te Ol kanes 


0 otherwise. (9) 


Note that the first line of (9) corresponds to the longi- 
tudinal individuals. The second line corresponds to the 
in-migrants selected through the Supplementary Sample. 
The third line represents altogether newborns, cohabitants 
(if the household is a longitudinal household not part 
of the Supplementary Sample) and/or initially-present 
individuals (if the household is part of the Supplementary 
Sample). 


The basic weight w; of household 7 is obtained from 


Mj 
/ 
dy Wik re 
= 1 : 
WwW; = Se Wik: (10) 
M; ane 
Lik 
k=1 


and finally w;, = w; for k €i. 


Using the basic weights obtained from the Weight Share 
method, one can estimate the total Y= yy Mi vy, 
of the characteristic y measured at year 2. The estimator 
used is the one given by equation (1). Using the definitions 
of the initial weights and the basic weights, Y can be 
rewritten as 
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where zt = Y* fork €iwith Y* = y Mi, y,,/M*. Thus, 
estimator (11) is simply the sum of two Horvitz-Thompson 
estimators related to s“!) and s*™. As shown in Section 2, 
this estimator is unbiased for Y. 


3.2.3 Variance Estimation 


The variance formula for Y is provided by equation (6). 
However, assuming that the two samples s“!) and s*™ are 
selected independently, we have Var( Y) = Var(Z*"") + 
Var (Z* °?)), where each term has the form of equation (6). 
For SLID, this assumption of independance holds if the 
selection of the Supplementary Sample is done through LFS. 

Considering Z*"!), we can re-index the individuals to 
reflect the fact that the m“!) individuals were selected at 
year 1 through n“!) households. This gives 


(1) ” oD of 
ee is f 
- pee aD pai ah 
Ker * fat jar 
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" i 
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ADD i ah 
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since, by selecting complete households 7/') = x}') for 


j € 1. The variance Var (Z* “!)) is then directly obtained as 


a NO) yD at - af xf ) 
Var(Z*()) = Da ys =f AM Zh Dy 
[=1 I=! (13) 


Considering Z*, the individuals can also be re-indexed 
for consistency with Z*", although this modification has 
no effect on the form of Z*®. Following the same steps 
used for Var(Z*), Var(Z*™) is obtained as 


, n«(2) nye (2) (a7 {2 — ae 2) 74, (2)) by O) 
Var(Z*) = Dy 3 SO ® VEG AOUM 
I=1 I’'=1 MENS (14) 


where N*®) is the number of households of year 2 con- 
taining at least one in-migrant and Z7°) = a Cais 
The quantity Mj represents the number of in-migrants 
present in household /. 
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Finally, Var(Y) is simply given by 


(1) ,Q) 
N N 1 1 
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The variance (15) may be unbiasedly estimated using 
the following equation: 
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As SLID is in fact a sub-sample from LFS, the jack- 
knife variance estimator developed for LFS (see Singh 
et al. 1990) may also be used, with minor modifications. 
In general, the jackknife method works as follows: the 
sample first is divided into random groups (or replicates, 
according to the LFS terminology). Then, each random 
group r is removed in turn from the sample and a new 
estimate Yon of the total Y is computed. The different 
estimates Y(,, are finally compared to the original estimate 
Y to obtain an estimate of the variance Var(Y). For 
further details on the jackknife method in general, see 
Sarndal, Swensson and Wretman (1992). 

Recall that LFS is based on a stratified multi-stage 
design which uses an area frame. Within each first-stage 
stratum h, the random groups (or replicates) correspond 
basically to the primary sampling units (PSUs). To compute 
the jackknife variance estimate for the estimation of the 
total Y, the following formula can be used: 


<x be Rel A = 
Var,;(Y) = yy ea ye (Yim) — Y)?, (7) 


h h reh 


where R,, is the number of replicates in stratum A and 
Ycnr) is the estimate of Y obtained after replicate r in 
stratum h is removed. For LFS, both Y and Y(q,) are 
poststratified based on the integrated approach of Lemaitre 
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and Dufour (1987). The plan is to use the same post- 
stratification approach for SLID but this discussion is out 
of the scope of the present paper. 
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Weighting Schemes for Household Panel Surveys 


GRAHAM KALTON and J. MICHAEL BRICK! 


ABSTRACT 


Household panel surveys often start with a sample of households and then attempt to follow all the members of 
those households for the life of the panel. At subsequent waves data are collected for the original sample members 
and for all the persons who are living with the sample members at the time. It is desirable to include the data collected 
both for the original sample persons and for the persons living with them in making person-level cross-sectional 
estimates for a particular wave. Similarly, it is desirable to include data for all the households for which data are 
collected at a particular wave in making household-level cross-sectional estimates for that wave. This paper reviews 
weighting schemes that can be used for these purposes. These weighting schemes may also be used in other settings 
in which units have more than one way of being selected for the sample. 


KEY WORDS: Cross-sectional estimates; Fair share weighting; Multiplicity weighting; Panel surveys; Weight 


share method. 


1. INTRODUCTION 


National panel surveys of household economics have 
been mounted in many countries in recent years. The U.S. 
Panel Study of Income Dynamics (PSID), conducted by 
the Survey Research Center of the University of Michigan, 
began in 1968 and has been collecting data on an annual 
basis since that time (Hill 1992), and the British Household 
Panel Survey began in 1990 (Buck ef al. 1994). Similar 
household panel surveys are also in progress or are being 
planned in most other European countries. The U.S. 
Bureau of the Census started to conduct the Survey of 
Income and Program Participation (SIPP) in 1983 (Nelson 
et al. 1985; Kasprzyk 1988; Jabine et al. 1990; Citro and 
Kalton 1993), and Statistics Canada introduced the Survey 
of Labour and Income Dynamics (SLID) in 1994 (Lavallée 
et al. 1993). 

A common feature to most of these household panel 
surveys is that they start with a national sample of house- 
holds, and then follow all the members of those households 
for the life of the panel. Over the course of time, household 
compositions change in a variety of ways. Some members 
of original sampled households leave those households to 
set up on their own or to join other households, as, for 
example, when a daughter leaves her parental household 
to get married. New members may join original sampled 
households, as, for example, when an elderly parent moves 
in with the family of a child or when a child is born to a 
household member. In order to be able to describe the 
economic circumstances of sample members at different 
points of time, household panel surveys usually collect data 


not only for the sample members but also for the individuals 
living with the sample members at the particular point of 
time. Following Lavallée (1995), these individuals are termed 
cohabitants in this paper. In other literature, they are often 
called associated persons or nonsample persons. 

As the panel duration increases, the proportion of 
cohabitants in the sample at a wave rises. For example, in 
the 1984 SIPP panel, cohabitants comprise about 8.6 per- 
cent of the sample after one year and about 12.6 percent 
of the sample after two years (based on Table | in Kasprzyk 
and McMillen 1987). With a long-term household panel 
survey, the proportion of cohabitants becomes substantial 
after several years. The PSID, for example, defines sample 
members as all persons in the family units sampled in 1968 
who are still alive, all the children born to these original 
sample members since the start of the panel, and the 
children of such children. In addition, the PSID collects 
data on the cohabitants who are living with sample 
members at each individual wave of data collection. Of 
the 20,535 individuals in interviewed family units in 1992, 
41.2 percent were original sample members, 34.6 percent 
were the children of original sample members born since 
the start of the panel and children of such children, and 
24.2 percent were cohabitants (excluding the Latino sample 
that was added in 1990) (Hill 1995). 

This paper reviews methods of weighting the data 
collected from both sample persons and cohabitants in 
order to produce unbiased (or approximately unbiased) 
estimates of population parameters. In considering the 
analysis of a household panel survey, three different types 
of analysis may usefully be distinguished: 
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© Cross-sectional analyses of households at a particular 
point in time; 

e Cross-sectional analyses of individuals at a particular 
point in time; 

e Longitudinal analyses of individuals over a period of time. 


Weighting schemes for these three types of analysis are 
discussed in later sections. Longitudinal analyses of house- 
holds over a period of time are not treated here because 
of the problematic nature of this type of analysis caused 
by changes in household composition (see, for example, 
Duncan and Hill 1985). 

The weighting schemes used in household panel surveys 
need to account for the fact that households and individ- 
uals included in the survey at a particular wave may have 
more than one route by which they can be selected. Ata 
given wave a household and its members are included in 
the sample if any of the original households (i.e. , house- 
holds existing at the time of the initial selection) from 
which the current household has drawn members was 
selected. With the usual weighting approach, households 
are assigned weights inversely proportional to their joint 
selection probabilities, taking account of the different 
ways they can be selected. However, this approach cannot 
be applied with most household panel surveys because 
these joint selection probabilities cannot be determined. 
The alternative weighting approach reviewed here, termed 
by Lavallée (1995) the weight share method, avoids the 
need to know the joint selection probabilities of sample 
elements, but it introduces a random variation into the 
weights. Since this random variation results in a loss in 
precision of the survey estimates as compared with the 
inverse selection probability weighting scheme, this alter- 
native approach should be considered only for situations 
where the joint selection probabilities cannot be ascer- 
tained. This situation often applies in household panel 
surveys and also in a number of other sample designs 
where elements can be selected by different routes. 

In order to prepare for the discussion of weighting 
schemes for household panel surveys, the next section 
elaborates on the household changes that can occur over 
time, and the types of individuals involved. Sections 3, 4 
and 5 then discuss weighting schemes that may be used for 
the three different forms of analysis described above. 
These sections deal with weighting schemes for unequal 
selection probabilities, without the complications of 
adjustments for nonresponse and noncoverage. The 
discussion relies heavily on previous work by Ernst (1989), 
Gailly and Lavallée (1993), Huang (1984), Judkins ef al. 
(1984), Lavallée and Hunter (1992), and Little (1989). 
Section 6 then briefly reviews the issues involved in making 
adjustments to the weights to compensate for missing data 
arising from nonresponse and noncoverage. Section 7 
presents some concluding remarks, and provides an illus- 
tration of another application of the weight share method. 


2. CHANGES IN POPULATION 
AND HOUSEHOLD 
COMPOSITION OVER TIME 


In analyzing a panel survey, it needs to be recognized 
that survey populations change over time. With household 
panel surveys it is important to distinguish between changes 
in population composition and changes in household 
composition. 

The composition of a survey population changes over 
time because some individuals leave the population, some 
enter the population, and some may leave and join the 
population more than once. Individuals leave the popula- 
tion through death, emigration, or entering an institution 
(for surveys of the noninstitutional population). They 
enter the population through birth (or reaching the specified 
minimum age), immigration, and leaving an institution. 

Households change composition over time for many 
different reasons, including deaths, births, marriages and 
divorces. For example, a household at time 1 may contain 
several individuals who end up in a number of different 
households at time 2. These individuals may set up new 
households on their own, they may join individuals who 
were in one or more households at time 1, or they may join 
individuals who were not in the population at time 1. One 
or more of the individuals may leave the population during 
the intervening period. 

Consider a simple sample design in which households 
are selected independently at time 1 with equal probability. 
At time 2, the sample of households comprises all the 
households that contain one or more individuals from the 
households sampled at time 1, and the sample of individ- 
uals at time 2 comprises all the members of the sampled 
households at time 2. The samples of households and 
individuals at time 2 are selected with unequal probabil- 
ities. For instance, the selection probability of a household 
at time 2 that contains individuals from three households 
at time | is three times greater than that of a household 
at time 2 that contains individuals from only one household 
at time 1. Similarly, the individuals in that household have 
three times the probability of selection. Thus weighting 
schemes that compensate for these unequal selection pro- 
babilities are needed for the analysis of the resultant data. 

Changes in population composition occur when individ- 
uals leave or enter the population. An individual sampled 
at time 1 who leaves the population before time 2 reduces 
the sample size for time 2 but does not otherwise affect 
cross-sectional estimates at time 2. In essence, the sampling 
frame for the time 2 population is the time 1 population, 
with the leavers in the intervening period being treated as 
blanks on the frame. Simply omitting the selected blanks 
from the time 2 sample causes no bias in the survey esti- 
mates (see, for example, Kish 1965). The situation with 
regard to entrants is, however, less straightforward. The 
household panel survey enumeration rule described above 
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incorporates new entrants who join households that contain 
individuals who were eligible for the initial sample into the 
population for cross-sectional estimates for later time points. 
However, new entrants who set up their own households are 
not represented in person-level analyses at later waves of the 
panel. Equally, households composed of only new entrants 
are not represented in household-level analyses at later waves. 

The failure of household panel surveys to cover house- 
holds composed of only new entrants presents a problem 
for cross-sectional analyses of later waves of the panel. If 
these households and their members constitute a negligible 
proportion of the population, the solution may be to 
simply ignore the problem. However, if the proportion is 
appreciable, as can occur in later waves of a long-term 
panel, alternative solutions may be called for. One possi- 
bility is to add a supplementary sample of new entrants 
(e.g., immigrants) to the panel, as discussed by Lavallée 
(1995) for the SLID. This solution is, however, often 
impracticable. Another solution is to limit the population 
of inference to persons who were members of the popula- 
tion at the start of the panel. New entrants found living 
with sample members are then excluded from the sample. 
This solution provides a clearcut definition of the popu- 
lation of inference. Whether the solution is appropriate 
depends on whether that definition can adequately satisfy 
the survey objectives. 

Changes in population composition pose problems for 
longitudinal analyses of individuals. For many purposes, 
the population of inference is restricted to those who were 
present in the population throughout the time period of 
observation specified for the analysis. The inclusion of 
cohabitants in longitudinal analysis also creates problems. 
If the time period for the longitudinal analysis starts at the 
beginning of the panel, the analysis can be restricted 
straightforwardly to original sample members. If the time 
period starts later, it is tempting to include both original 
sample members and cohabitants joining the panel before 
the start of the analytic time period. However, the usual 
enumeration rules for household panel surveys specify that 
data are collected for cohabitants only while they continue 
to live with original sample members, that is, they are not 
followed if they cease to live with such persons. Unless the 
time period is short enough that the number of cohabitants 
who cease to live with sample persons in that period is 
negligible, this enumeration rule makes it problematic to 
include cohabitants in longitudinal analyses. This problem 
is discussed further in Section 5. 


3. CROSS-SECTIONAL ESTIMATES 
FOR HOUSEHOLDS 


This section considers weighting schemes that may be 
used to produce cross-sectional estimates for households 
for any wave of a household panel survey after the first. 
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At the first wave a sample of households is selected and 
all the individuals in the sampled households become panel 
members to be followed throughout the life of the panel 
or until they leave the survey population. At a subsequent 
wave, wave f, the household sample comprises all the 
households in which panel members reside. Households 
that consist of new entrants only are not represented in the 
sample at later waves. Such households are ignored here. 
Complications of nonresponse are deferred until Section 6. 

Consider the estimation of the total Y for all H 
households in the population at time fr: 


H 
a De Yin (3.1) 
A general estimator for this total can be expressed as 
H 
Y = 3 W; Yen 


where w; is a random variable that takes the value w; = 0 
if household jis not in the sample. The expectation of Yis 


H 
E(Y) = ») E(w,) Y;. (3.2) 


i=1 


By comparing equations (3.1) and (3.2), it can be seen that 
Y is unbiased for Y for any weighting scheme for which 
E(w;) = 1 for all i. 

There are many ways to satisfy the condition E(w;) = 1. 
Three will be treated here. First, consider a standard inverse 
selection probability weighting scheme. The probability 
of a household being in the sample at time ¢ is the proba- 
bility of one or more of the households at time 1 from 
which it has drawn members being selected for the original 
sample. The probability of household H; being in the 
sample at time ¢ is then 


P(H,) = P(h;Uh,URU...) 
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where P(h; Uh, Uh, U.. .) is the selection probability of 
the union of original households h;, hy, hy, etc. for the 
original sample, p; is the selection probability of original 
household h; for the original sample, pj, is the joint selec- 
tion probability of original households h; and h, for the 
original sample, efc. and where households h;, hy, hp, 
etc. each contain at least one member who is currently in 
household H;. The weight for each sampled household is 
then w; = 1/P(H;). With this weighting scheme, 
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satisfying the condition for an unbiased estimator of a 
population total. 
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In practice, the computation P(A) will generally not 
be as complex as equation (3.3) might suggest because the 
number of original households represented in household 
H; is usually small. With, say, two original households 
involved, P(H;) reduces to 


P(H;) = P(A, Uhy) = py + D2 — Pr. (3.4) 


A problem with the application of the inverse selection 
probability approach is that p; may be known only for 
households selected for the original sample, and not for 
other households. Also the joint probability may not be 
known. Even when the original sample was selected with 
equal probabilities, so that all the p; are the same, the 
joint probability may depend on the sample design (for 
instance, whether the two households were in the same 
segment or not). The difficulty of obtaining P(H;) is a 
major drawback with the inverse selection probability 
approach. 

An alternative strategy for developing the weights for 
time ¢ is to base them only on the selection probabilities 
of households selected for the original sample, thus avoiding 
the difficulty in obtaining P(H;) noted above. One 
approach is to identify the set of households h; at time 1 
that would result in household H; being in the sample at 
time ¢, and compute the weight for household AH; as 


dj 


where w; = 1/p;if household h,, which has at least one 
member in household H;, was selected for the original 
sample and w; = 0 if not, and where aj; are any set of 
constants satisfying };a, = 1. 


With this approach, 
E(w) = pjU/p;) + (1 — p,)0 = 1, 
and hence 


E(w) = Y) ay = 1. 


J 


Thus, the use of weights w; will yield unbiased estimators 
of totals for the household population for any choice of 
constants aj, provided that ); aj = 1. As indicated 
above, the principal advantage of this type of scheme is 
that it requires information only on the initial selection 
probabilities of the original households that were sampled 
at time 1, which are known. It does not require information 
on the initial selection probabilities of the other original 
households that have members in the current household, 
which are often not known. 

A natural choice of a; is to make them equal for all 
the original households that lead to the selection of house- 
hold H; at time ¢. Huang (1984) terms this scheme a 


multiplicity approach. Here the scheme will be called an 
equal household weighting scheme. With this scheme 


w; = a wi/Cj, (3.6) 
J 


where C; is the number of original households represented 
in household H; at time ¢. 

An alternative version of the above approach is one 
based on original sample persons rather than households. 
In this case, let J;;, denote individual k from original 
household / in household 7. Then 


_— /, 
wi = YD un Mies 
iwik 


where wj, = 1/p, if individual k in household h; was in 
the original sample and w;, = 0 if not, and where the 
aij, are any set of constants satisfying Y; Yay, = 1. 
Since the probability of an individual being selected for 
the original sample is the same as that of that individual’s 
household, 


E(wix) = pj/pj) + Cl — pj)0 = 1. 


In this case, the natural choice of the constants aj, is 
to make them equal for all members of the current house- 
hold who were eligible for selection for the original sample. 
This produces what has been termed the fair share 
weighting scheme (Huang 1984; Ernst 1989). This scheme 
is termed here an equal person weighting scheme. With this 
scheme 


where w;; = w;, is constant for all individuals in house- 
hold H; emanating from the same sampled household at 
time 1, M;; is the number of individuals in household H; 
coming from household h;, and M; = ¥};Mjis the number 
of individuals in household H; who were eligible for the 
sample at time 1. The equal person weighting scheme is 
applied in the SIPP and is proposed for use in the SLID. 

Although developed here in terms of persons rather 
than households, it is readily apparent that the equal 
person weighting scheme could equally have been generated 
in terms of households. As shown above, the household 
weight w; = )); aj; satisfies the condition E(w;) = 1 
for any set of constants aj such that };aj; = 1. The 
equal household weighting scheme chooses aj = 1/C;, 
with ));aj = 1. The choice aj = Mj/M;, with Yaz = 1, 
leads to the equal person weighting scheme. 

It is instructive to compare the inverse selection probab- 
ility weighting scheme with the equal household and equal 
person weighting schemes in a simple case. Following 
Little (1989), consider household H; selected at time ¢ 
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with household members coming from two original house- 
holds. Let p; and p, denote the selection probabilities for 
the original households, and let p,;, denote their joint 
selection probability. Under the inverse selection probab- 
ility approach, the household weight is 


1 
4 Se ee 
OV sw [Dy = BO 


as indicated above. 


Under the equal person weighting scheme the weight for 
household H; depends on which household or households 
were selected for the original sample: 

w; = P,/p, if only household h, was selected; 
w; = P,/p> if only household h, was selected; 
= (P;/p,) + (P2/p2) if both h, and hy were selected; 


= 
| 


where P, and P, are the proportions of members of house- 
hold H; who came from households h, and hp, respectively 
(excluding any new entrants to the population). The probab- 
ility of only household h, being selected is (p,; — pj2), of 
only household h being selected is (py — pj7), and of 
both households being selected is p,,. The expected value 
of the weight conditional on household H; being in the 
sample is thus 


E(w; | H; in sample ) = 


(D1 —P12) (P1/p1) + (P2- P12) (P2/p2) +Pi2l (P1/p1) + (P2/p2) | 
Pi + Po — Pi2 


> 


Lex 


1 
E(w; | 4;.in sample). = —————_———__ = w}. 
es ee 


As this result demonstrates, the weight for household H; 
varies depending on which original households were 
selected, but in expectation the weight is the same as that 
obtained from the inverse selection probability approach. 

Results for the expectation of the weight of household 
H; under the equal household weighting scheme can be 
readily obtained as a special case of the above derivation 
in which P; = P, = %. In expectation, the weight is the 
same as that for the inverse selection probability approach. 

Given that the weight w; = ); a, w;; satisfies the con- 
dition E(w;) = 1 for any set of a such that Yjaj = 1, 
the question arises as to the optimal choice of the a. One 
approach is to choose the a; to minimize the variance of 
the estimated total Y. 


The variance of Y may be expressed as 


V(Y) = VE(Y | s) + EV(Y |), (3.7) 
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where s denotes the set of households in the sample at 
time ¢. Now 


H 
E(Y | s) =£(¥ w¥.1 8) 


i=1 


AY Ss 
ye E(w; | Hi) Y; = ya wPY; = Y*, 


where Y* is the standard inverse selection probability 
estimator. Thus 
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The first term in equation (3.7) is thus the variance of the 
standard inverse selection probability estimator, and the 
second term is the additional variance resulting from the use 
of weighting schemes from the class (3.5), w; = Yjajw 
The aj; may then be chosen to minimize EV( Yosh 


v3 wa 13) 


AY 
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Consider 


V(¥ | s) 


)s yy Y,;Y;,Cov (w;, w;}’ | H;,—7;'). 


ini’ 
Assuming Cov(w;,w; | H;,H;-) = 0, 
V(Y¥ |'s).= yy Y¥?7V(w; | Hi) 


since, as noted above, E(w; | H;) = w;*. Thus, assuming 
Cov(w;,w; | H;,H;-) = 0, V(¥ | s) is minimized when 
E(w? | ;) is minimized. 

Consider the application of this approach to the simple 
case discussed above in which H; is composed of members 
from two original households and let w; = aj;wj/, + 
(1 — a;) Wj. Then 


EO yy 
2 2 2 
a; (1 —a;) a; 1l-a; 
(Pi Pin) 5 FO =P) > FP (2a) 
Pi P2 Lier P22, : 


P\ + P2- P12 
Minimizing E(w? | H;) is equivalent to minimizing 
A = (Pi — Pir)p3a? + (P2 — Pr)pi(. — a)” 


+ Pil (pr — Pi); + Pi) ?. 
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Then 


OA 
Sus = 2(p; — Py2)p3a; — 2(py — Pi2)pP (1 — a) 
Qj 
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Solving 0A/da; = 0 for a; gives the optimum a; as 


a —1 
ee (: ae ore (3.8) 
Hy = 12O 


If the original households are selected independently, /.e., 
[iy SS VOW aay. 


doi = Ler - f * ale 3.9) 


PiU — pp) 1 
where ¥; = p;/(1 — p;) is the odds of original household 
h; being selected. 
Irrespective of whether the households are sampled 
independently, in the special case of an equal probability 
(epsem) sample of households initially, with p; = po, 


oj > . 
Thus, in the two-household case, the equal household 
weighting scheme minimizes the variance of the household 
weights around the inverse selection probability weight 
when the initial sample is an epsem one. 

The optimal choice of a,; given by (3.8) requires knowl- 
edge of p;, p2 and pj, and that given by (3.9) requires 
independence and knowledge of p, and pp. If these pro- 
babilities were known, then the standard inverse selection 
probability weight could be employed and would be 
preferable. In the case of an approximately epsem sample, 
the equal household weighting scheme should be close to 
the optimal, at least for the case where the members of the 
household at time ¢ come from one or two households at 
the initial wave. This would apply, for instance, in the case 
of an epsem initial sample, with perhaps a few departures 
from epsem. With the equal household weighting scheme, 
when only one of the C; original households, h;, repre- 
sented in H; was selected for the original sample (as will 
generally be the case), then the weight for H; is simply 
1/Cip; : 

In the case of a non-epsem initial sample, the choice of 
the a; would ideally depend on the original household 
selection probabilities. However, since these probabilities 
are unknown, that approach cannot be applied. By default, 
the equal household or equal person weighting schemes 
may therefore be employed in this case. The use of these 
schemes (or any scheme with constant a,j’s satisfying 
) je; = 1) with a non-epsem initial sample still results in 


an unbiased estimate Y. The drawback to these schemes 
in such a case is only that the a;; are suboptimal in terms 
of minimizing the variance of Y. 

It should be noted that the equal household weighting 
scheme requires information on the number of original 
households A; contributing members to household H; at 
time ¢. That number may be difficult to determine in some 
cases. Consider, for example, a household at time ¢ that 
contains two cohabitants. It may sometimes be difficult 
to determine whether these two persons were in a single 
household or in two separate households at the time of the 
initial sample selection. The equal person weighting scheme 
has the attractive feature of avoiding the need for Wave 1 
household information, except for persons in sampled 
households at Wave |. This feature provides an important 
reason for preferring the equal person to the equal house- 
hold weighting scheme. 


4. CROSS-SECTIONAL ESTIMATES 
FOR INDIVIDUALS 


In producing cross-sectional estimates for individuals 
for any wave of a household panel survey after the first, 
it needs to be recognized that some new entrants will have 
joined the survey population since the start of the panel. 
New entrants who join households that contain one or 
more members of the original population can be repre- 
sented in cross-sectional estimates for later waves, but new 
entrants living in households that do not contain any 
members of the original population are not covered (unless 
a special sample of them can be taken). The former type 
of new entrants is included in the weighting procedure 
described below, but the latter type is not. 

Let there be N individuals in the population at time ¢, 
with WN; individuals in household H;(i = 1, 2, ..., A) 
and YN; = N. The members of household H; come from 
households h;, hy, hy, etc., at time 1. Let Mj; denote the 
number of members of household H; at time t who were 
in household h; at the start of the panel. The sum M = 
YY M,; is less than the population size at time 1 because 
of leavers from the population in the period from time 1 
to time f, and M < N because of new entrants to the 
population who are in households containing members 
from the original population. 

Consider now the estimation of a total for the population 
of individuals at time fr: 


Z 


r=¥ 


i= 


Vie (4.1) 
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where Yj, is the value for individual k in household H;. 
As in the household case discussed in the previous section, 
a general estimator for this total can be expressed as 
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Ni 
Y=) OU WHY, (4.2) 
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where wij, is a random variable that takes the value w;, = 0 
if individual k in household H; is not in the sample. The 
estimator Y is unbiased for Y provided that E(w;,) = 1 
for alli and k. 

As noted earlier, there are many ways to satisfy the 
condition E(w.) = 1. It is instructive to consider three 
of them. First, let w,, = 0 for all individuals not in the 
original sample. In this case, the estimator Y discards 
cohabitants. Let p;, denote the probability of a member 
of the original population, individual k residing in house- 
hold H; at time ¢, being selected for the initial sample, and 
let w;, = 1/pj,. Then, for such an individual 
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With this scheme, all new entrants to the population have 
wi, = 0 with certainty. Thus Y in (4.2) provides an 
unbiased estimator of the total for the original population 
that is still present at time ¢, but does not include a com- 
ponent for the new entrants. 

Modifications to the above procedure can be made to 
cover certain types of new entrants. For instance, births 
to sampled mothers can be included by assigning them the 
weights of their mothers, or if, as in the SIPP, the survey 
population is taken to be adults aged 16 and over, those 
under 16 at the start of the panel can be treated as sampled 
persons with assigned probabilities, and they can be 
included in the analyses of later waves after they have 
attained the age of 16. Such modifications do not, how- 
ever, handle all types of new entrants. Provided that the 
proportion of other types of new entrants is small, this 
deficiency may not be a serious concern. 

The weighting scheme that restricts the analysis to 
original sample persons, plus certain specified new entrants, 
is employed with the PSID. Its limitation is that it fails to 
make direct use of data collected for cohabitants. Such 
data may be used to provide information on the situation 
of sample persons, but the cohabitants are excluded from 
the sample for the analysis. 

In order to include cohabitants in cross-sectional 
analyses for time ¢ they need to be assigned positive weights. 
Noting that the probability of an individual being selected 
for the sample is the same as that of his or her household, 
weighting schemes for cross-sectional analyses of indi- 
viduals at wave ¢ can be obtained directly from those for 
households given in Section 3. Here we will develop the 
general strategy of producing weights for cross-sectional 
analysis at time ¢ based only on the selection probabilities 
of members of the original sample, thus avoiding the 
problems with the inverse selection probability approach 
noted in Section 3. 
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Let /;;, denote individual k from original household h; 
who is now in household H;. Let w; denote the weight for 
every member of household H; for cross-sectional analyses 
at time ¢, and let 


Wie s SS ijk Wijk 
yack 
where wjj, = 1/p; if household A; was in the original 
sample and w;, = Oif not. Then, as before, E(wj,) = 1 
for members of the original population. New entrants, for 
whom p; = 0, may be handled by setting a; = 0. Then 
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provided that );Y,ajjx = 1. Under this condition Y is 
unbiased for Y. 

A natural choice of ajjx is to set aj, = 1/M,; for all 
members of the original population. This is the equal person 
weighting scheme in which every member of household H; 
at time ¢ (including new entrants) receives the weight 


Ww; = ye yo Wijx/M;j. 
prekk 


Another choice of the aj, is that used for the equal 
household weighting scheme. Let C; denote the number 
of original households that have members in household 
H at time ¢. Then YY aj = 1 can be divided equally 
between households, with each member of original house- 
hold h; being assigned a value of aj, = 1/C;M;. Then 
for original household h; 


Mij 
2 Aik = ICs 
k 


The derivation of the a;;, to minimize the variance of 
the estimated total Y for the population of individuals 
follows directly from the corresponding derivation for the 
population of households given in Section 3. The estimated 
total for the population of individuals is 


A Ss Nj S Ni 
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since the weights for every individual in sampled household 
H, are the same. This estimated total can be expressed as 


n = y wiY;, 
i 


where Y; = YY is the household total for H;. Thus Y 
can be expressed as a household total, and the results of 
Section 3 can be applied directly. 
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Consider the example from Section 3 in which H; is 
composed of members from only two original households, 
perhaps together with one or more new entrants. In this 
case the person-level weight w; = )j Lx ojjxWijx reduces to 


i (z ca) wi + (x cs) Wi? 
k k 


= ajwa + (1 — aj)wWa, 


= 
II 


where a; = Y,ajjx. AS shown in equation (3.8), the 
optimum value of a; is 


ons =I 
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The individual values a,j, are not needed for computing 
the w,; only the original household totals Y,ajj, are 
required. If individual values are needed for the aj, they 
may be simply assigned as Y),ajj./M;j. 

As in the household case, the optimum weighting a,; 
requires knowledge of p,, p> and p),. If these probabilities 
are known, the standard inverse selection probability 
weight w; can be computed, and would be preferred. In 
the case of an approximately epsem sample, the equal 
household weighting scheme should fare well. However, 
the equal household weighting scheme requires informa- 
tion on the number of original households contributing 
members to current household H;, and this information 
may not always be available. As discussed in Section 3, 
for this reason the equal person weighting scheme may be 
preferred. 


5. LONGITUDINAL ANALYSES OF 
INDIVIDUALS 


A key analytic advantage of a panel survey is the ability 
to conduct longitudinal analyses relating variables for the 
same sampled units measured at different time points. 
Since all persons in original sampled households are 
followed throughout the life of the panel or until they leave 
the survey population, the data they provide may be 
readily analyzed longitudinally for any time period within 
the panel’s time span (although nonresponse adjustments 
may be needed for panel attrition). Thus, for example, in 
a ten-year panel, data for original sampled persons may 
be analyzed from year 1 to year 10, from year 5 to year 
9, or for any other period. New entrants (e.g., births) may 
be included in the analysis for periods beginning after the 
start of the panel provided that they are treated as panel 
members who are followed throughout the panel even 
when they leave the households of original sampled 
persons. 


Given the weighting schemes described in the previous 
section, cohabitants can be included in cross-sectional 
analyses of later waves. These weighting schemes provide 
a cross-sectional representation of the population at any 
wave of the panel (apart from new entrants not living 
with original population members). It is then possible to 
consider all the sample of original sample members and 
cohabitants at time ¢ as the initial sample of a new panel 
that may be used for longitudinal analyses from time ¢ to 
(t + k). This procedure is, for instance, used in the SIPP, 
where all original sample members and cohabitants present 
at the start of the second year of the panel are included 
in analyses relating to that year. 

The limitation to the inclusion of cohabitants in longi- 
tudinal analysis is that the following rules used in most 
household panel surveys specify that cohabitants are 
dropped from the panel if they cease living with original 
sample persons. Thus, cohabitants who live with original 
sample members at the start of the analysis period but who 
cease to live with them before the end of that period effec- 
tively become nonrespondents. If the analysis period is 
relatively short, the number of such nonrespondents may 
be small and the risk of serious nonresponse bias may be 
negligible. If the analysis period is a long one, however, 
the number of not-followed cohabitants may be appre- 
ciable, causing concerns about potential bias. The issue 
here is one of a trade-off between the reduced variance due 
to the increase in sample size from including cohabitants 
in the analysis versus the increased bias resulting from 
the additional nonresponse caused by failing to follow 
cohabitants leaving the households of original sample 
persons. 

The additional nonresponse bias can be avoided by 
changing the following rules to specify that cohabitants 
are to be followed from the time they join the panel for 
the rest of the life of the panel, or until they leave the 
survey population, irrespective of whether they continue 
to live with original sample members. This change, how- 
ever, leads to an expanding panel and the need for addi- 
tional resources. Not only do data need to be collected for 
cohabitants at waves after they cease to live with sample 
persons, but data also need to be collected for any persons 
with whom the cohabitants live at later waves. 


6. ADJUSTMENTS TO COMPENSATE 
FOR NONRESPONSE AND 
NONCOVERAGE 


The discussion thus far has assumed that data are 
collected for all sampled persons and their cohabitants and 
that all the target population is covered by the sampling 
procedures. In practice both these assumptions are violated. 
Nonresponse is present in nearly all surveys and is of 
particular concern in household panel surveys, where some 
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sampled households fail to respond at the initial wave and 
others fail to respond at some of the subsequent waves. 
The sampling frames used in most surveys are subject to 
some degree of noncoverage, and in later waves of house- 
hold panel surveys there is an additional source of non- 
coverage associated with new entrants to the population 
who are not living with members of the original population. 

In a simple cross-sectional survey, missing data can be 
classified into item nonresponse, total nonresponse and 
noncoverage. Imputation procedures can then be used to 
assign values for item nonresponses, weighting adjustments 
can be applied to compensate for total nonresponse, and 
poststratification adjustments can be applied to compen- 
sate for nonresponse and noncoverage. The situation is 
made far more complex in panel surveys by the occurrence 
of wave nonresponse, which arises when a sampled element 
responds for some but not all of the waves for which it was 
eligible. Not only do methods need to be devised to com- 
pensate for wave nonresponse, but also the preferred 
methods of compensation may depend on the type of 
analysis to be performed, in particular whether cross- 
sectional or longitudinal analyses are to be conducted. 

From one perspective wave nonresponse can be viewed 
as a set of item nonresponses in the element’s longitudinal 
record, suggesting that imputation may be used to fill in 
the missing values. Alternatively, it can be treated as total 
nonresponse, handled by weighting adjustments. The 
imputation approach is more natural for the creation of 
a panel file for longitudinal analysis, whereas the weighting 
approach is more natural for the creation of a cross- 
sectional file for the analysis of the data collected at a 
single wave. 

The attraction of the imputation approach with a longi- 
tudinal file is that it retains all the reported data, whereas 
the weighting approach discards the reported data for all 
the elements that fail to provide data for one or more 
waves for which they were eligible. However, the imputa- 
tion approach may involve the fabrication of a large 
amount of data, especially when an element fails to respond 
at several waves. Thus, for panel files, a compromise solution 
may be preferred, imputing responses for elements with 
few missing waves and using weighting adjustments to 
compensate for those with several missing waves (including 
total nonrespondents). In the SIPP, for example, imputa- 
tion is used to assign responses for sample persons with 
a single missing wave that is bounded on both sides by 
responding waves, and weighting adjustments are used for 
all other sample persons with missing waves (Singh ef al. 
1990). Further discussion of methods of handling wave 
nonresponse in panel files is provided by Lepkowski 
(1989), Kalton (1986), and Lepkowski et a/. (1993). 

Another complication of some household panel surveys 
is the occurrence of partial household nonresponse, which 
occurs when the survey data are collected for some but not 
all members of a sampled household at a particular wave. 
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The lack of data for one individual in a household means 
that key household characteristics (e.g., household earnings) 
cannot be computed. One solution is to drop the household 
and its responding members from the sample, and use a 
weighting adjustment. Another is to impute the responses 
for the nonresponding household members, as is done in 
SIPP (where they are termed Type z nonrespondents). With 
the latter solution, data are available for all members of 
responding households, and hence person-level adjustments 
are unnecessary within responding households. 

We now turn to consider the issues involved in dealing 
with missing data for cross-sectional analyses of a house- 
hold panel survey. A separate cross-sectional file containing 
data for all responding households and their members 
(either deleting the households or imputing values for 
missing responses in the case of partial household non- 
response) can be created for each wave. Adjustments are 
then needed to compensate for the nonresponding and 
noncovered households and persons in each file. 

Nonresponding households at wave ¢ can be divided 
into total nonrespondents and wave nonrespondents. 
Total nonresponse occurs in a panel survey when a sampled 
element fails to provide data for any wave. Since it is 
common practice not to follow up sample households that 
fail to respond at the initial wave, these households and 
their members are generally the total nonrespondents. 
Compensation for total nonrespondents is relatively 
straightforward. The Wave | weights of the responding 
households at the initial wave can be adjusted using standard 
nonresponse adjustment methods and the adjusted weights 
can be used instead of the selection probabilities in devel- 
oping the cross-sectional weights for later waves. Most 
nonresponse adjustment methods, such as weighting class 
adjustments (Kalton and Kasprzyk 1986) and adjustments 
based on response propensities (Little 1986), are based on 
the assumption that nonresponse is random within 
weighting classes or that the probabilities of responding 
within a class can estimated precisely. Under these condi- 
tions, the response mechanism can be treated as an addi- 
tional stage of sampling. Thus, the selection probabilities, 
p;, used to define the weights in equation (3.5) may be 
redefined as the product of the selection probabilities and 
the adjustment due to nonresponse. For example, if 
weighting class adjustments are used, the selection proba- 
bility of original household h; multiplied by the weighted 
response rate for the weighting class in which h; falls is 
used instead of the original p;. The previous results then 
follow for the weights adjusted for total nonresponse. 

The same approach can also be extended to cover 
weighting adjustments for households responding at the 
initial wave that lead to no responding households at wave 
t. In this case, the responding households at the initial 
wave can be divided into weighting classes based on 
responses given at that wave, and the weights of households 
leading to one or more responding households at wave f 
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can be further adjusted to compensate for those leading 
to no responding households at wave ¢. The revised wj; 
can then be employed in equation (3.5) and subsequently. 

Both the above nonresponse adjustments are applied 
in relation to the original households. A further type of 
household nonresponse cannot be handled in this way. 
This type of nonresponse involves the situation where an 
original household splits into two or more separate house- 
holds at wave f, and where some but not all of those 
households respond at that wave. In this case the adjust- 
ment for the nonresponding households needs to be made 
in relation to the wave ¢ households, H;, rather than the 
original households, h;. If the number of original house- 
holds having members in each wave ¢ nonresponding 
household of this type were known, the weights w; for 
these households could be computed using the approach 
described above. Then weighting adjustments could be 
readily applied within weighting classes of the wave ¢ 
households to compensate for the nonresponding house- 
holds. In practice, however, the number of original house- 
holds having members in a nonresponding household at 
wave f may often be unknown. One approach for handling 
this situation is to estimate this number by the average 
number for responding households at wave ¢ that have 
similar characteristics to (e.g., they are also splits from 
original households), and are in the same weighting class 
as, the nonresponding household. Using such estimated 
numbers where necessary, the weights w; can be deter- 
mined for all nonresponding households of the type being 
discussed. Standard weighting adjustments can then be 
applied to the responding households at wave ¢ to com- 
pensate for these nonresponding households. 

Incomplete coverage of the target population is another 
nonsampling problem that has been traditionally addressed 
in surveys by adjusting the sampling weights. For example, 
poststratification (see, for example, Holt and Smith 1979) 
and generalized raking procedures (Deville et a/. 1993) are 
often used to adjust the weights so that they sum to counts 
from independent sources not subject to undercoverage. 
These adjustments may also reduce the sampling errors of 
the estimates, although bias reduction is often more critical. 

The control totals used in most household surveys are 
counts of the number of persons in classes defined by 
characteristics such as age, sex and race. This method of 
reducing undercoverage bias may be fully sufficient when 
estimates of persons are the only types of statistics to be 
produced from the survey. However, further steps are 
needed to calculate household-level weights for producing 
statistics of household characteristics. 

One approach to developing household-level weights 
when control totals are based on person-level counts is 
called the principal person method, as described by Alex- 
ander (1987). In this method, poststratification adjustments 
are applied at the person level. One household member is 
then identified as the principal person and the fully adjusted 


weight for that person is assigned to be the household 
weight. Since the person weights are already adjusted to 
the control totals, the household weight does incorporate 
some adjustments to reduce coverage bias. For cross- 
sectional estimation from a household panel survey, the 
principal person method can readily be used in conjunction 
with the equal household and person weighting schemes 
to produce household level weights. 

A disadvantage of the principal person method is that 
estimates of the number of persons calculated using the 
principal person weight will generally differ from control 
totals. The estimates may also differ significantly depending 
on the criteria used to identify the principal person in the 
household. The method has also been criticized because 
the weights of the members of the same household differ, 
even though they were all selected at the same rate as the 
household. Estimation schemes proposed by Alexander 
(1987), Lemaitre and Dufour (1987), and Zieschang (1990) 
address these objections by constraining the household 
weights so that they are consistent with the independent 
person-level totals while minimizing the distance between 
the original household weights and the adjusted weights. 
All three consider variants of a generalized least squares 
(GLS) algorithm to achieve this objective. Zieschang 
(1990) shows how GLS can be used to create weights that 
are consistent with the person controls and force all persons 
within a household to have the same weight. 

The application of GLS methods when the household 
weights are computed using the equal household and 
person weighting schemes is relatively straightforward. 
However, empirical evaluation of the consequences of 
using these methods is needed. The GLS methods have the 
unattractive feature that they can result in negative weights. 
Furthermore, the increase in the variation in the weights 
arising from the constraints imposed may result in less 
precise estimates. This concern may be especially important 
when the variability in the household weights is increased 
due to their multiple routes for selection and the equal 
household or person weighting schemes are necessary. 


7. SUMMARY AND CONCLUDING REMARKS 


This paper has described weighting schemes for cross- 
sectional analysis of later waves of a household panel 
survey using data for all households for which and all 
individuals for whom data are collected. These weighting 
schemes can accommodate new entrants to the population 
who move in to live with members of the original popula- 
tion, but not other new entrants. 

The usual inverse selection probability weighting scheme 
requires information on the household selection probabil- 
ities of all members of the households sampled at a later 
wave, as well as the joint selection probabilities of the 
original households that contribute members to the later 
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wave households. The inverse selection probability weighting 
scheme can often not be applied because these probabilities 
are unknown. To deal with this problem, an alternative 
approach that requires information on only the selection 
probabilities of sampled original households is described. 

This alternative approach produces a class of weighting 
schemes including the equal person (fair share) scheme 
used in SIPP and the equal household weighting scheme. 
All the schemes in this class produce weights that are in 
expectation equal to those produced by the usual inverse 
selection probability scheme. The variance in the weights 
around the inverse selection probability weights gives rise 
to an increase in the variance of the survey estimates. 
When the original households are selected with approx- 
imately equal probability, the equal household weighting 
scheme is near optimal for both household and individual 
level analyses to control this increase in variance. 

The alternative class of weighting schemes produces 
unbiased estimates of population totals for any choice of 
constant a; that satisfies the condition Y};a;; = 1 and for 
any initial sample design. The equal household and equal 
person weighting schemes are, however, suboptimal for 
non-epsem initial samples. One of them may nevertheless 
be the appropriate scheme for such designs, because the 
optimal choice of the a; depends on the unknown initial 
selection probabilities, and hence cannot be determined. 
The equal household and equal person weighting schemes 
have different data requirements, in that the former 
requires knowledge of the number of Wave 1 households 
represented in the Wave f household whereas the latter 
does not. The fact that this information may not always 
be readily obtainable thus argues in favor of the equal 
person weighting scheme. 

The cross-sectional individual weights for a particular 
wave can be used as the starting weights for a longitudinal 
analysis that begins at that wave. This procedure includes 
cohabitants present at that wave in the longitudinal 
analysis. However, if cohabitants are not followed when 
they cease to live with sampled persons, those who leave 
sample persons before the end of the period of the longitu- 
dinal analysis become nonrespondents. Before cohabitants 
are included in a longitudinal analysis, a check should 
therefore be made to ensure that their inclusion will not 
give rise to risks of serious nonresponse bias. 

The class of weighting schemes described has a broader 
range of application than that indicated here. It can in fact 
be usefully applied in any situation where an inverse selec- 
tion probability weighting scheme would be appropriate, 
but where not all the inclusion probabilities and joint 
inclusion probabilities are known. Consider, for instance, 
the modified version of the Mitofsky-Waksberg random 
digit dialing sampling procedure for telephone surveys 
described by Brick and Waksberg (1991). A sample of 
telephone numbers (primes) is selected at the first stage of 
this two-stage sample design. If a prime number is found 
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to be a working residential number, that household is 
selected and a fixed number of additional telephone 
numbers in the same 100-bank is selected. The households 
found at these numbers are then all included in the sample. 
If a prime number is not a working number, the sampling 
process stops. With this procedure, the probability of a 
working residential number being selected depends on the 
number of working residential numbers in its 100-bank, 
and hence differs across 100-banks. This probability can 
be estimated from the sample of telephone numbers in the 
100-bank. A complication arises, however, when a sampled 
household has two or more telephone numbers. In this 
case, the selection probability of the sampled telephone 
number can be estimated, but those of the nonsampled 
numbers cannot. Thus, the standard inverse selection 
probability weighting scheme cannot be used. However, 
the alternative weighting scheme described here can be 
employed. 
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Modelling Net Undercoverage in the 1991 
Canadian Census 


PETER DICK! 


ABSTRACT 


In 1991, Statistics Canada for the first time adjusted the Population Estimates Program for undercoverage in the 
1991 Census. The Census coverage studies provided reliable estimates of undercoverage at the provincial level and 
for national estimates of large age - sex domains. However, the population series required estimates of undercoverage 
for age - sex domains within each province and territory. Since the direct survey estimates for some of these small 
domains had large standard errors due to the small sample size in the domain, small area modelling techniques were 
needed. In order to incorporate the varying degrees of reliability of the direct survey estimates, a regression model 
utilizing an Empirical Bayes methodology was used to estimate the undercoverage in small domains. A raking ratio 
procedure was then applied to the undercoverage estimates to preserve consistency with the marginal direct survey 
estimates. The results of this modelling process are shown along with the estimated reduction in standard errors. 


KEY WORDS: Small area; Empirical Bayes; Undercoverage. 


1. INTRODUCTION AND BACKGROUND 


The Census of Canada is conducted every five years; 
one of its objectives is to provide the Population Estimates 
Program with accurate baseline counts of the number of 
persons by age and sex within each province and territory. 
Unfortunately, not all eligible persons are correctly enu- 
merated by the Census. As part of the evaluation of the 
Census, Statistics Canada estimates, through two sample 
surveys, the net number of persons missed by the Census. 
The estimates are from the Reverse Record Check Study, 
which estimates the gross number of persons missed by the 
Census, and the Overcoverage Study, which estimates 
persons double counted or erroneously included in the 
final Census count. When combined the figures estimate 
the net number of people missed by the Census. 

These surveys were designed to produce reliable direct 
estimates for large areas, such as provinces, and for large 
domains, such as age - sex combinations at the national 
level. However, the Population Estimates Program requires 
estimates of missed persons for single year of age for both 
sexes for each province. However using the direct survey 
estimate would result in individual estimates having unac- 
ceptably high standard errors due to insufficient sample 
in the small domain. One approach to reducing the variance 
of the small domain estimates would be to borrow strength 
from related domains. This approach leads to creating an 
explicit model for the small domain that can be used to 
predict the net missed persons in that domain. 

The result of modelling the small domain estimates is 
to produce a series of estimates with a smaller Mean Square 
Error than the direct estimate. However, as opposed to the 


direct survey estimate which is design unbiased, the 
modelling approach will introduce a bias for each estimate. 
Thus modelling the small domain estimates implies that 
a trade off is required between reducing the variance of 
each estimate and the bias introduced through the modelling 
process. One approach to ensuring that the more reliable 
direct survey estimates are utilized is to introduce an 
Empirical Bayes model. This procedure creates an estimate 
that is a combination of a model estimate and the direct 
survey estimate weighted by their respective variances. It 
is an Empirical Bayes estimate instead of a Bayes estimate 
because underlying parameters are first estimated, then 
these estimated parameters are considered known in later 
calculations. Note that since the individual sampling 
variances are used in the estimation, a more precise direct 
estimate would contribute much more to the final Empirical 
Bayes estimate than a similar estimate with low precision. 
This ensures that the model does not dominate estimates 
that are already considered reliable. It is also possible to 
approach this estimation problem through a Hierarchical 
Bayes methodology: details on this method can be found 
in Datta, et al. (1992). Ghosh and Rao (1994) give an 
appraisal of both the Hierarchical Bayes and Empirical 
Bayes approaches to small area estimation. 

Outside of Canada, two different approaches to 
smoothing the Census undercoverage have been described 
in the literature. In the United States, the net undercoverage 
in the 1990 American Census was evaluated by means of 
the Post Enumeration Survey (Hogan 1992). Initially, it 
was planned to multiply the US Census counts by adjust- 
ment factors (the ratio of true population over the enu- 
merated population) for 1,392 a priori defined post strata. 
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These estimated adjustment factors would then be used 
to adjust the Census count for missed persons. Since some 
of these 1,392 estimated adjustment factors had high 
standard errors, it was proposed to smooth the direct 
estimates through an Empirical Bayes regression model, 
similar to one proposed by Ericksen and Kadane (1985), 
and then to rake the smoothed estimates to agree with direct 
estimates for large geographic regions. However, this 
approach was criticized by Freedman and Navidi (1992). 
Eventually, the United States Department of Commerce, 
the U.S. Census Bureau’s parent agency, decided not to 
proceed with adjusting the Census counts for under- 
enumeration in July 1991. Consideration was also given 
in the United States to adjusting the post Censal popu- 
lation estimates for undercoverage in the Census, but the 
Department of Commerce also rejected this adjustment. 

The Australians use a different method than the 
Americans for estimating the domain totals. Choi, Steel 
and Skinner (1988) describe a methodology that incor- 
porates the estimates of net undercoverage of the Census 
into the population estimates but leaves the actual Census 
counts as enumerated. The under enumeration is estimated 
through a Post Enumeration Survey (PES) and demo- 
graphic analysis. The small domain estimates are produced 
by raking the Census age counts for each sex to the PES 
estimates for national age/sex totals and part of State/ 
Territory/sex totals. 

The procedure proposed for the 1991 Canadian Census 
combines some of the elements of both the American and 
Australian approaches. As in the American procedure, a 
model is postulated for the underlying true adjustment 
factors and another model is postulated for relating the 
direct survey estimates to the true underlying adjustment 
factors. Through Empirical Bayes estimation, a new 
smoothed adjustment factor is estimated that will havea 
lower MSE then the direct survey estimate. These smoothed 
adjustment factors are then converted into estimates of 
missed persons. The Australian method for constraining 
the resulting estimates to known marginal totals is then 
adopted. These final raked estimates are used as the base 
for the small domain estimates of missed persons. In turn, 
these estimates are adjusted to account for known demo- 
graphic principles (See Michalowski 1993). Details on the 
technical criteria for adjustment of the population esti- 
mates can be found in Royce (1992). 

This paper is organized as follows. In Section 2, some 
background information on the two sample surveys is 
described and the basic Empirical Bayes model is presented. 
Assumptions and limitations of the model are also discussed 
and the estimation of the parameters is briefly discussed. 
In Section 3, the explanatory variables used in the regres- 
sion model are presented and the model building process 
is described. The final model is presented and the results 
displayed. Section 4 presents a discussion on the rationale 
behind constraining the Empirical Bayes estimates to 


reliable marginal totals. The final adjusted estimates are 
then presented. Finally, Section 5 presents some conclusions 
and topics for further study. 


- 2. MODEL FOR THE ADJUSTMENT FACTORS 


2.1 Background and Notation 


The model for the adjustment factors requires input 
data. The actual data originates with two coverage studies: 
the Reverse Record Check (RRC) and the Overcoverage 
Study (OCS). The RRC is used to estimate the number of 
persons missed by the Census while the OCS is used to 
estimate the number of persons erroneously included in 
the Census count. These surveys are designed to give 
reliable estimates of net undercoverage for all provinces, 
some of the larger metropolitan areas and for some large 
national domains, such as males aged 20 to 24. Since the 
surveys are independent, it can be assumed that the 
variance of net missed persons will be the sum of the two 
estimated variances from the RRC and the OCS. Further 
details on these studies can be found in Germain and Julien 
(1993) and the 1991 Census Technical Report - Coverage 
(Statistics Canada 1994). 

The domains of interest can be defined by partitioning 
the sample into p = 1,2, ..., Pprovinces/territories and 
a= 1,2,...,A age -sex groups, henceatotalofA x P 
domains require estimates. Let C; be the number of 
persons in the i-th province - age domain enumerated in 
the Census and 7; be the true population of the same 
domain. The net number of persons missed in the /-th cell 
is M; = T; — C;. The adjustment factor, 9,, is the ratio 
of the true population in a domain over the Census count, 
while the undercoverage rate, U;, the unit that is usually 
reported in the releases from the coverage studies, is the 
ratio of missed persons over the true population. 

The true adjustment factors, 8;, which are the variables 
that we wish to estimate, can be written as: 


TE REE 


0; = 
C; C; 


Undercoverage rates (U;) which are usually reported 
in the releases from Statistics Canada, are related to the 
adjustment factors through the relationship 


U; = M;(M; + Cana = 1 = Qt: 


In the modelling of the adjustment factors, the creation 
of ultimate domains is required. These domains are those 
at which the actual direct survey estimates of the adjust- 
ment factors will be produced. There must be an estimate 
for each province (10) and territory (2), so immediately P 
is fixed at 12. The age groups were fixed at 4 to create 
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national estimates that have acceptably low standard 
errors. These age groups are defined for male and female 
as follows: 0 to 19 years of age; 20 to 29 years of age; 30 to 
44 years of age; and 45 years and older. In total there are 
12 xX 8 = 96direct survey estimates of adjustment factors 
that have to be fitted into the Empirical Bayes model. Each 
domain requires, besides the direct estimate of the adjustment 
factor, an associated estimate of the sampling variance. 


2.2 Model and Assumptions 


The basic model for the undercount is composed of two 
distinct parts. The first part describes how the direct survey 
estimates are related to the true underlying adjustment 
factors, while the second part models the relationship 
between the true adjustment factors and a set of explanatory 
variables. Since the parameters in the regression model are 
estimated by first estimating the parameters of an assumed 
underlying prior distribution and then assuming that these 
estimated parameters are known for any further calcula- 
tion, this model is known as an Empirical Bayes model 
(Maritz and Lwin 1989). 

The first part of the model, the sampling model, relates 
the observed adjustment factors to the true adjustment 
factors. This relationship is assumed true within each 
domain, and can be expressed as: 


the observed adjustment factor = 
the true adjustment factor + a random error. 


The sampling model is written as follows: 


F, = 90; + €;: €; ~ Normal (0,07), 
lala) ae ie aX eles 


where 9; is the true adjustment factor and €; is a random 
error component with a variance of 07. The assumptions 
underlying this model are: 


(a) the sampling errors, €;, have mean zero; 


(b) the sampling variances, oF are known in each of the 
n domains; 


(c) since the sample was selected independently within 


each domain, the covariance between the sampling 
errors €; in domain / and €; in domain / is zero; and 


(d) the random errors €; are normally distributed in each 
domain. 


Further discussion on the assumption of the known 
sampling variance in each domain is given below. 


The second part of the model, the regression model, 
relates the true adjustment factors to a set of underlying 
explanatory variables. This model states that: 


the true adjustment factor = a linear combination 
of explanatory variables + a random error. 
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The regression model can be written as: 


0; = X; B te 6; : 6; ~ Normal (0,77), 
(i = lly sy soon it SAL xKye 


where X; is the i-th row in X, a known (n X p) matrix of 
explanatory variables, Bisa (p X 1) vector of unknown 
regression parameters and 6; is (a different) random error 
with a model variance of 7”. Underlying the system model 
are the following assumptions: 


(a) the model errors, 6;, have mean zero; 
(b) the model variance, 7’, is constant over all n domains; 
(c) the model errors, 6;, are normally distributed; 


(d) the model errors, 6;, are independent of sampling 
ChEOLS Ges 


(e) the covariance between different domains is zero 
(i.e., Cov(6;, 6;) = 0). 


The problem is to use both the sampling model and the 
regression model to estimate 9;, the true adjustment 
factors. The conditional expectation for 9; given 6, Ors 
7, F; can be determined for the joint model. Using stan- 
dard arguments (Rao 1973), it can be shown that the 


conditional expectation of 9; is: 
E(; | 6,07,77,F;)) = (1 — w)X;8 + o)F;, (1) 


where w; = TT of eae 


Equation (1) is the basis for all the estimates that follow, 
although a few modifications need to be made before 
applying it to the data. Note that it is basically a weighted 
average of the direct survey estimate and the regression 
model estimate of the adjustment factor. Each estimate 
is weighted according to the precision with which it was 
estimated. If the sampling error, 07, is small compared to 
the model error, 7”, implying that the direct survey esti- 
mate is relatively precise, then the final smoothed estimate 
will be mainly composed of the direct survey estimate. 
However, if the direct survey estimate has a large sampling 
variance relative to the model variance then the final 
smoothed estimate will be mainly constituted from the best 
linear unbiased predictor. The amount each estimate con- 
tributes to the final smoothed estimate is controlled by the 
weighting coefficient, w;. 

Some limitations apply to interpretations that can be 
made about this model. First, it must be emphasized that 
this model is purely descriptive; it cannot be considered 
to be a causal model. Since the primary goal of this model 
is descriptive, the inferences on the regression parameters, 
B, while interesting are not of primary importance. Hence, 
the final regression model when it contains a term, say, 
on British Columbia renters and not Manitoba renters, 
is only saying that British Columbia renters explain a 


Variance of missed persons 
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significant portion of the variation in adjustment factors 
in British Columbia while Manitoba renters does not 
explain a significant portion of the variation in adjustment 
factors in Manitoba. 

As mentioned above, the sampling variances associated 
with the direct survey estimates of the adjustment factors 
are considered known in the Empirical Bayes model. 
However, experience has shown that the directly estimated 
variances are, in fact, somewhat unstable. In order to 
create some stability with the estimation of these variances 
it is proposed to model them. If we consider the design of 
the two sample surveys, then, under relatively mild assump- 
tions, Dick (1993) has shown that within each domain the 
variance of the estimate of missed persons is proportional 
to a power of the Census count. If we add in appropriate 
normalizing parameters, then this relationship can be 
written as: 


iis OF = V(M;) = KG. 
or, as in the form of a regression equation, 
Log(V(M;)) = a + ylog(C;) + yn; with 


ni ~ N(0,§?). 


@ Observed ——Estimate 


1,000,000,000 
100,000,000 
10,000,000 
1,000,000 
100,000 
10,000 


1,000 


1,000 10,000 100,000 


1,000,000 


Census count 


Figure 1. Observed variance vs. census. 


This model for the sampling variance assumes that the 
product of the design effect and the undercoverage rate 
is constant within each domain. As discussed in Dick 
(1993), this assumption appears to be reasonable. Figure 1 
shows the plot of the observed variance of missed persons 
calculated from the two coverage studies versus the Census 
count for the 96 domains. The least squares regression line 
was estimated as 


log(9(M;)) = — 6.133 + 1.715 logC, 


10,000,000 


and is also plotted in Figure 1. A residual analysis (Dick 
1993) did not detect any apparent violations of the under- 
lying model assumptions. Since, in addition, the coefficient 
of determination, the R’, is 0.943, this model was adopted 
for producing the sampling variances. The estimated survey 
variances were calculated for the adjustment factors through 


9(F;) = 0(M))/C?. 


It will be assumed that these predicted values for the 
sampling variances are the actual ‘known variances’ 
required for the Empirical Bayes model. 


2.3 Parameter Estimation 


So far the model has been described in purely Bayesian 
terms: only the parameter 9; is considered unknown. 
Taking the usual Empirical Bayes approach (Maritz and 
Lwin 1989), we will assume that all the parameters except 
G, the regression parameter, are known. The conditional 
expectation of 0; with the regression parameter estimated 
can be written as 


PN E(O; | Prost he) 


However, in practice, the model variance, 7’, is also 
unknown and must be estimated. The conditional expec- 
tation of 0; will now change to 


Fie) =) E(6; | B, 0}, ja Fs), 


where the sampling variance, 07, is still considered known. 


To estimate the model variance and the regression coef- 
ficients in the Empirical Bayes model, the marginal distri- 
bution of the observed adjustment factors, m(F;) ~ N 
(x; B, 7? + 07), can be used. Three possible methods were 
examined for estimating the variance parameter, 77, : 
Method of Moments (MM) as in Fay and Herriot (1979), 
Maximum Likelihood (MLE) as in the PES in the United 
States (Hogan 1992) and Restricted Maximum Likelihood 
(REML). 

It is well known that MLE estimation of variance 
components is biased downwards (Harville 1977). Under- 
estimation of the model variance in the Empirical Bayes 
model would result in more reliance being placed upon the 
regression model instead of the direct survey estimate. This 
is aresult we wished to avoid. In Dick (1993), it is shown 
that there is little difference between the estimates of the 
model variance from REML or MM. Since the REML has 
a well understood asymptotic theory, it was adopted for 
the estimation of the model variance in the Empirical 
Bayes model. 

Harville gives a full account of REML estimation. The 
basic approach is to first estimate the regression parameter, 
and then to estimate the model variance from the resulting 
residuals instead of the actual data. If we let X* be a matrix 
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of (mn — p) linear contrasts such that E[X*‘F] = 0, 
then Harville shows that the resulting (log) likelihood 
function, L,em;, when maximized with respect to the 
unknown model variance will give the restricted maximum 
likelihood estimates. 

In the context of the Empirical Bayes model, Harville’s 
approach can be described as follows. First, an initial esti- 
mate, usually zero, of the model variance, tt), is made 
and then the regression parameter, 8, is estimated through 
weighted least squares: 


Bay SIV eX Vek, (2) 
where Vo = diag (779) + o7:i = 1,4.., n). Using this 
estimate of 6), a new REML estimate of the model 
variance, 7tys can be made through 


OL 
as A ae [eass) alge) aa ak, = ON okt.) 
T 


where, if we set P, = Vi! —VoOUX(Xt' VOIX)! XV>}, 
we have 


OB al pe sist Pale & (Ei XB) Vial hus XB) 
ar? 2 K > K K 
and 
O7L. 1 51 B 
i(7?) = —E a = — trace (P' P.). 
d(d77) 2 


Note, upon convergence of 7” and 8, i(7”) ~! will be the 
asymptotic variance of 77. 

By iterating between (2) and (3), new estimates of 77 
will be used to update the estimate of 8, which in turn will 
be used to update the estimate of 7”. The iterations then 
continue until a suitable convergence has been reached: 
in this case ((724,/72) — 1) < 10~° was used. 

Once the estimates for @, the regression parameters, and 
77, the model variance, have been determined, then the 
final smoothed estimates can be found. Maritz and Lwin 
(1989) show that the Empirical Bayes, or smoothed, 
estimate can be written as 


Fe = (1 — &)X;6 + oF, 


where @; = 77(77 + 07) ~!. This is a combination of the 
original estimate and the regression estimate weighted by 
their respective variances. 

The objective of the smoothing model is to create a 
series of estimates with smaller MSE than the original 
estimates. Prasad and Rao (1990), through asymptotic 
arguments, have suggested using the following estimator 
for the mean square error: 
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: " du; \? 
MSE [F¢?] = MSE[F%"] + te w; E(#? — rl, 
iT 


The mean square error for the Empirical Bayes estimate, 
using restricted maximum likelihood estimation, has been 
conjectured by Cressie (1992) to be: 


MSE[F*] = MSE(F%) + 2 8;(72) = 
8(77) + &)(77) + 2 8(77), 
where 


8(77) = 7(1 — &)) 


A 


8>;(77) = (1 — &))* XP (Xt V- LX) 1X; 
and 
83(77) = (1 — &)?(# + of) [i741 71. 


The assumed normality of €; and 6; is an important 
assumption in the derivation. Note the value for the 
sampling variance, o?, is assumed known. 

Prasad and Rao give the following interpretation to 
each of the three components: g,;(77) is the Bayes estimate 
of the variance, £;(77) is the contribution from estimating 
the regression parameters and §;;(77) is the contribution 
from estimating the model variance 7”. An estimate of 
the component due to the estimation of the sampling 
variance is not available: the additional variance this 
would add is not known but its absence clearly implies that 
the MSE is underestimated. 


3. EMPIRICAL BAYES LINEAR MODEL 


3.1 Explanatory Variables 


The Empirical Bayes model described above was fitted 
to the 96 observed adjustment factors, with the sampling 
variances estimated using the method described in Section 2. 
The linear model that was fitted to this data included the 
following explanatory variables: 

(a) An indicator variable for each province/territory. 

(b) An indicator variable for each sex. 

(c) An indicator variable for each age group. 

(d) A variable indicating the percentage of people in the 
domain that are renters. 

(e) A variable indicating the percentage of people in the 
domain that do not speak either official language. 

(f) Various interaction variables including province by 
renters, province by non-official language, age and sex 
by renters. 

In total, 42 variables were used in the initial regression. 
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These variables were selected for the initial regression 
model based, in part, on the experiences of previous RRC 
studies (Burgess 1988), partly on the results of the 1991 
coverage studies (Germain and Julien 1993) and partly on 
the experiences of the PES in the United States as described 
in Hogan (1992) and Datta et al. (1992). The actual ratio- 
nale for the variables to be included are as follows: 


(a) The province indicator was included as an indication 
of the difficulty of Census collection within each 
province. Prior to the 1991 Census, it was assumed that 
collection would be more difficult in British Columbia 
and Ontario, and the anecdotal field evidence during 
collection seemed to support this conjecture. 

(b) The age and sex variable were included because of the 
known differences in undercoverage rates between 
males and females. The undercoverage, in previous 
studies, has also shown a marked increase for indi- 
viduals in their 20’s. 

(c) Tenure, in effect the percent of renters in each domain, 
was included because of the experiences in the United 
States PES, results of previous RRC studies and as a 
suggestion from the Statistics Canada Statistical 
Methods Advisory Committee. 


(d) The use of non-official language was an attempt to 
locate the immigrant and minority groups that in the 
past have tended to have higher undercoverage rates. 

(e) The interaction terms were included to further refine 
the predictive power of the model. 


The mean encompasses all those variables that are not 
included in the model. Note that since indicator variables 
are used for province, sex and age-sex, one variable has 
to be excluded in order to avoid a singular design matrix. 
In effect, the missing variable, say the province indicator 
for Newfoundland, is included in the mean. 

An operational constraint was also placed on the model. 
The SAS IML program written to estimate the parameters 
was limited to 4,095 numeric elements in the design matrix, 
hence with 96 domains, or observations, the model was 
limited to a maximum of 42 variables. 


3.2 Model Building Process 


After starting with the full regression model and 42 ex- 
planatory variables, a procedure was needed to remove 
those variables that were not statistically significant. The 
procedure chosen was to eliminate the least significant 
variable after each completed estimation cycle. This 
implies that for the 42 variable model, the variable Female 
Renters aged 0 to 19 would be eliminated since it has a 
t-value of 0.05. The regression model was then re-run with 
the remaining 41 variables. The least significant variable 
was then eliminated from that model. This procedure is 
equivalent to the Backward Stepwise Regression described 
in Draper and Smith (1966, page 167). 


The Backward Stepwise Regression method was used 
to eliminate all variables until all remaining variables had 
a ¢-values greater than 2 (in absolute value). However when 
the final model was examined, it was noticed that a multi- 
collinearity problem existed between the indicator variables 
for certain provinces and the interaction terms for renters 
within the same provinces. The implication of this problem 
is that there are some explanatory variables which are highly 
correlated with each other. This in turn implies that not all 
parameters in the model can be estimated precisely. As a rule 
of thumb Judge et a/. (1984, page 459) suggest that this can 
be a problem when the simple correlation between variables 
is greater than R?, the coefficient of determination. The 
final model had a R* = 0.85 and the simple correlation 
between the variables in question were all greater than 0.90 
(in absolute value, since the correlations were negative). 

A solution to this problem was to delete the variables with 
the lower ¢-values which turned out to be the provincial indi- 
cators. The final model is shown in Table 1 with the estimated 
coefficients and their ¢-values. The effect of removing the 
provincial indicators was to lower the final R” from 0.85 
to 0.844, thus little predictive power has been lost. 


Table 1 
Final Estimates of Variables Used in Regression 
Final T-Value 
Category Variable Estimate ae ae 
(8)  (Ho:6 = 0) 
Mean Mean 1.0075 $75.72 
Agemses Male 20 to 29 0.0563 15.34 
Gambnanon Male 30 to 44 0.0208 5.81 
Female 20 to 29 0.0240 6.49 
Sex by Age by Female 
Non-Official Language 
Language 0 to 19 0.0797 DAD 
British Columbia 
Renters 0.0449 3.96 
Ontario Renters 0.0804 7135 
Tenure.by Quebec Renters 0.0255 2.66 
Province New Brunswick 
Renters 0.1064 5.61 
Yukon Renters 0.0639 3.80 
Northwest Territories 
Renters 0.0682 6.22 


The final regression model then had various diagnostic 
tests performed on it. Since the regression is a weighted 
least squares with a random error term, Lange and Ryan 
(1989) have suggested using the following form to create 
standardized residuals: 


The residuals were analyzed using both Q-Q plots and 
outlier detections methods: no major departures from the 
assumed distribution of the residuals were detected. More 
details on the residual analysis can be found in Dick (1993). 


Survey Methodology, June 1995 Sil 
Table 2 
Direct, Smoothed and Raked Estimates of Adjustment Factors 
Sex Age Estimate B.C. Alta Sask. Man. Ont. Que. N.B. N.S. 12a Bia Nfld Yukon N.W.T. 
Male 0-19 Direct 1.017 1.026 1.012 1.029 1.028 1.017 1.022 1.019 1.004 0.999 1.031 1.036 
Smooth 1.019 1.013 1.009 1.013 1.029 1.016 1.027 1.010 1.007 1.006 1.026 1.027 
Raked 1.020 1.016 1.011 1.015 1.031 1.018 1.027 1.013 1.005 1.007 1.029 1.031 
20-29 Direct 1.087 1.036 1.068 1.058 1.113 1.071 LE 122 1.063 1.060 1.057 1.098 1.127 
Smooth 1.086 1.056 1.065 1.062 1.104 1.074 1.103 1.064 1.063 1.062 1.094 ei22 
Raked 1.083 1.061 1.073 1.067 1.101 1.079 1.096 1.073 1.041 1.074 1.096 1.127 
30-44 Direct 1.031 1.021 1.028 1.034 1.054 1.047 1.043 1.018 1.025 1.026 1.069 1.080 
Smooth 1.039 1.026 1.028 1.030 1.053 1.041 1.046 1.026 1.028 1.028 1.052 1.059 
Raked 1.038 1.028 1.032 1.032 1.051 1.043 1.043 1.029 1.018 1.033 1.053 1.059 
45+ Direct 1.019 1.018 1.002 1.014 1.013 1.011 1.014 1.016 1.018 1.016 0.992 1.076 
Smooth 1.017 1.011 1.006 1.009 1.019 1.013 1.019 1.010 1.009 1.009 1.021 1.039 
Raked 1.014 1.010 1.006 1.009 1.016 1.012 1.015 1.010 1.005 1.010 1.019 1.035 
Female 0-19 Direct 1.034 1.018 1.017 1.012 1.037 1.029 1.029 1.014 0.995 1.016 1.026 1.054 
Smooth 1.030 1.015 1.013 1.015 1.038 1.023 1.030 1.010 1.006 1.010 1.028 1.061 
Raked 1.032 1.018 1.016 1.017 1.040 1.026 1.030 1.012 1.004 1.013 1.030 1.068 
20-29 Direct 1.068 1.047 1.028 1.020 1.072 1.043 1.070 1.030 1.004 1.041 1.068 1.072 
Smooth 1.058 1.036 1.031 1.029 1.070 1.044 1.071 1.031 1.027 1.033 1.069 1.092 
Raked 1.058 1.041 1.036 1.032 1.070 1.048 1.068 1.037 1.018 1.041 1.072 1.099 
30-44 Direct 1.013 1.009 1.004 1.006 1.027 1.017 1.031 1.019 1.004 1.024 1.031 1.020 
Smooth 1.018 1.008 1.007 1.007 1.030 1.017 1.029 1.010 1.007 1.011 1.028 1.026 
Raked 1.017 1.008 1.007 1.007 1.028 1.017 1.025 1.011 1.004 1.012 1.027 1.026 
45+ Direct 1.007 1.003 1.018 1.001 1.011 1.011 1.000 1.002 0.993 1.013 1.024 1.007 
Smooth 1.014 1.006 1.010 1.006 1.021 1.015 1.020 1.006 1.005 1.009 1.031 1.026 
Raked 1.008 1.004 1.007 1.004 1.012 1.009 1.011 1.004 1.002 1.006 1.019 1.016 


3.3 Estimates of Adjustment Factors 


Table 2 shows both the direct survey estimate and the 
smoothed Empirical Bayes estimate of the adjustment 
factors. An inspection of the table shows that these esti- 
mates are relatively close, reflecting the Empirical Bayes 
methodology of combining the direct survey estimate with 
the model estimate. Note that all of the domains that were 
originally estimated to have overcoverage —- shown by an 
estimated adjustment factors being less than one - have 
been changed, by the Empirical Bayes estimates to being 
an estimate of undercoverage. The difference between the 
two sets of estimated adjustment factors - in absolute 
terms - differ by under 1% and in the larger provinces by 
less than 0.5%. However, for some of the smaller provinces 
and territories the difference between the two estimates can 
be substantially larger. In the Northwest Territories the 
change between the directly estimated adjustment factor 
and the Empirical Bayes estimate is about 2% for 3 age - 
sex groups and over 3% for another. 

The objective of the Empirical Bayes model is to produce 
estimates with smaller MSE than the survey estimates. 
From Section 2.2 it can be shown that the variance for the 
direct survey estimates is calculated from 


logi(F;) = — 6.133 — 0.285 logC,, 


while the Prasad-Rao MSE, from Section 2.3, is calcu- 
lated by 


MSE[F¥?] = MSE(F%*) + 283;(77) = 


8, (77) + Bj(77) + 283;(77). 


Ratio ( Prasad-Rao / Survey) 
1 
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Alta. Man. 


Yukon 
Nfld. N.W.T. 


Figure 2. Ratio of root MSE, Prasad-Rao and survey. 
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Figure 2 plots, for each domain, R = (MSE [Fe] /$(F;)) 
the ratio of the root mean square errors for the Empirical 
Bayes model and the estimated survey variance (Note that 
within provinces the domains are ordered as Male aged 
0-19, 20-29, 30-44 and 45 and over and Female aged 0-19, 
20-29, 30-44 and 45 and over). Clearly, the Empirical Bayes 
MSE is smaller in all domains. However, in the larger 
provinces, Ontario and Quebec, the ratio of the root MSEs 
is only between 0.7 and 0.8. This relatively small gain is 
a reflection of the large sample sizes in these domains which 
in turn give a reliable estimate of the variance. The large 
gains are made in the smaller provinces and territories. For 
instance, in Prince Edward Island, the ratio of the root 
MSEs are all smaller than 0.5 showing the large improve- 
ment in the estimates. The one outlier is in the Northwest 
Territories (females aged 0 to 19): the Prasad-Rao MSE 
appears to have been overestimated in this domain. 


4. ADJUSTMENTS MADE TO EMPIRICAL 
BAYES ESTIMATES 


4.1 Rationale and Methodology 


The advantage of the Empirical Bayes method is 
apparent from the above discussion. However, the 
Empirical Bayes methodology does not preserve the higher 
level (i.e., the large domain) direct survey estimates that 
are reliable. By this it is meant that the provincial totals 
and the age - sex domain totals for the direct survey 
estimates and the Empirical Bayes estimates are not equal. 
Since the two surveys were designed to produce estimates 
at these levels, it is crucial that the Empirical Bayes be 
consistent with these reliable marginal totals. 

To achieve consistency of estimates of missed persons 
between the reliable provincial and age - sex totals from 
the direct survey estimates and the final Empirical Bayes 
estimates, a raking ratio procedure was used. This is basi- 
cally the method used in Australia to determine their small 
domain estimates (see Choi ef al. 1988). This technique 
re-scales the individual Empirical Bayes estimates to 
conform to the known provincial and national age - sex 
totals. Once this procedure has converged, the final esti- 
mates will be consistent with the direct survey totals. In 
terms of a log-linear model, we are using as the main 
effects (province and age-sex) estimates the results from 
the two coverage studies and the interaction terms (province 
by age-sex) estimates from the Empirical Bayes modelling. 

Details of the procedure can be described as follows. 
Assume that we have a matrix of estimated missed persons 
that has P columns (corresponding to the provinces) and 
A rows (corresponding to the age-sex groups). First set 
Fy, = F;, then let Mia = Cya(Foa — 1) be the direct survey 
estimate of the number of missed persons in province p and 
age - sex group a and let M&” = M® = C,,(F&” — 1) 
be the Empirical Bayes estimate of missed persons from 


the Empirical Bayes model. If we let a plus sign (+) 
represent addition across the variable then the raking esti- 
mate can be written for cycles k = 0, 1, ... as; 


A A 
mage = (YY Meal Ya) 
a=li 


a=! 


and 


P P 

A (2K+2 A (2k+1 y 7 (2k+1 

mee = ngs? Sat! Ye me”). 
p=1 p=1 


This procedure will converge to a unique solution. Since 
this is basically a log-linear model, the underlying assump- 
tion is that the relationship determined by the Empirical 
Bayes model for the interaction between province and 
age - sex group is valid and will be preserved. 


Change (Empirical Bayes / Raked) 


-4.0 -3.0 -2.0 -1.0 0 1.0 2.0 3.0% 
Change (Direct / Empirical Bayes) 


Figure 3. Percent change in estimates of adjustment factors. 


Table 2 shows the final raked estimates of the adjust- 
ment factors along with both the original survey estimates 
and the Empirical Bayes estimates. Generally, the impact 
of raking is to shrink the Empirical Bayes estimate back 
towards the survey estimate. This is shown in Figure 3. 
Here two different percent changes in the estimated adjust- 
ment factors are plotted. The X-axis shows the percent 
change between the direct survey estimate and the Empir- 
ical Bayes estimate. The Y-axis shows the percent change 
between the Empirical Bayes estimate and final raked 
estimate. The plot shows that the two variables are nega- 
tively correlated: hence the raking tends to move the 
Empirical Bayes estimates closer to the original survey 
estimates. 


Survey Methodology, June 1995 


One draw back of this procedure is that the MSEs of 
the raked adjustment factors are now very difficult to 
estimate. Due to the non-linear nature of the raking ratio 
procedure, a direct calculation is impossible. It is possible 
to use a Taylor series expansion; however this assumes a 
large sample size in each domain when in fact we know 
some domains have very small sample sizes. A possible 
procedure is to adjust the estimated MSE from the Empir- 
ical Bayes estimates and multiply these by the squared ratio 
of the raked Empirical Bayes estimate over the Empirical 
Bayes estimate. While this procedure is only a crude 
approximation, it can at least give some guidance as to the 
reliability of the individual estimates. This method will 
ensure that the coefficient of variations calculated for the 
Empirical Bayes estimates will be retained for the corre- 
sponding raked Empirical Bayes estimates. This is the 
procedure that was used to produce the final MSE esti- 
mates for the raked Empirical Bayes estimates of missed 
persons. 


4.2 Detailed Domain Estimates 


The Population Estimates Program requires even finer 
detail than that produced by the various models discussed 
above. In fact the program needs estimates for single years 
of age for each sex for each Census Division within each 
province. Since the Empirical Bayes methodology is limited 
somewhat by the direct survey results - an estimate with 
a non-zero standard error is required for each domain — 
synthetic methods must be used to generate the more 
detailed estimates. 

For the Population Estimates Program, estimates for 
each province and sex were produced for 9 age groups 
instead of the 4 age groups used in the Empirical Bayes 
model. A straight synthetic model, using the raked Empir- 
ical Bayes estimates as initial values, was proposed for this 
stage of estimation. To produce these more detailed esti- 
mates, the raked Empirical Bayes estimate was allocated 
proportionally by Census count across all sub-age groups 
within each province and sex. Let the final raked estimate 
in the p-province and the a-th age-sex group be M?s*? = 
Mg. Also if the a-th age - sex group is composed of Q 
exclusive sub-age groups then the estimate of the missed 
persons in the p-th province and the g-th sub-age group 
within the a-th age - sex group would be 


where Cpa, = Cpa = g=1Cpa,- This approach guarantees 
that the estimates from the earlier raked Empirical Bayes 
output are preserved for the original domain total. The 
further estimates that are required for the population 
estimates program use demographic methods. In fact, one 
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of the objectives of the Empirical Bayes procedure is to 
provide initial estimates for the demographic methods. See 
Michalowski (1993) for further details. 


5. SUMMARY AND CONCLUSIONS 


The Empirical Bayes methodology was adopted because 
it preserves the more reliable estimates from the larger 
provinces and domains while permitting a model based 
estimate to dominate if the underlying direct estimate is 
unreliable. This is in accordance with standard survey 
methods of using the direct survey estimates as much as 
possible. The raking ratio procedure used for adjusting the 
estimates from the Empirical Bayes model was used to 
ensure consistency with the direct survey results that were 
known to be reliable. 

As for the explicit model used to describe the underlying 
true adjustment factors, it must be noted that this model 
is purely descriptive. Its primary function is to use explana- 
tory variables to describe the variation in adjustment 
factors, taking into account the sampling error associated 
with each adjustment factor. It would not be prudent to 
make far-reaching conclusions on the nature of under- 
coverage from the final set of parameters included in the 
model. 

The main weakness of this approach is with the two 
variances that are estimated. The assumption of the regres- 
sion model errors being approximately normally distributed 
is difficult to assess. In the absence of any real knowledge 
about the true underlying distributions any assumption 
about the model variance will be essentially unverifiable. 
The proposed model variance seems reasonable and 
diagnostic checks have not revealed any major problems. 

The sampling variance model is more problematic. All 
Empirical Bayes methods assume that this variance is 
known, when in fact it has to be estimated. Efforts to 
extend the Prasad-Rao MSE calculation to include the 
contribution from this estimated parameter have not 
yielded any new results. 

In the future, research will concentrate in working 
around the problem associated with estimating the sampling 
variances. Further work needs to be conducted on the 
Prasad-Rao MSE calculation. In addition, the possibility 
of using the micro level data from the coverage studies and 
estimating the undercoverage rates directly through logistic 
regressions as in Wong and Mason (1985) will be pursued. 

Another project would be to examine the implications 
of recasting the Empirical Bayes model into the standard 
state space framework (Robinson 1991). Pfeffermann and 
Burck (1990) have suggested a method for calculating the 
MSE for a time series placed in a state space model that 
has to conform to certain periodic benchmarks. The state 
space formulation would also be useful in explicitly incor- 
porating the demographic methods. 
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Between-State Heterogeneity of Undercount Rates and Surrogate 
Variables in the 1990 U.S. Census 


JAY JONG-IK KIM, ALAN ZASLAVSKY and ROBERT BLODGETT! 


ABSTRACT 


As part of the decision on adjustment of the 1990 Decennial Census, the U.S. Census Bureau investigated possible 
heterogeneity of undercount rates between parts of different states falling in the same adjustment cell or poststratum. 
Five ‘‘surrogate variables’”’ believed to be associated with undercount were analyzed using a large extract from the 
census and significant heterogeneity was found. Analysis of Post Enumeration Survey on undercount rates showed 
that more variance was explained by poststratification variables than by state, supporting the decision to use the 
poststratum as the adjustment cell. Significant interstate heterogeneity was found in 19 out of 99 poststratum groups 
(mainly in nonurban areas), but there was little if any evidence that the poststratified estimator was biased against 
particular states after aggregating across poststrata. Nonetheless, this issue should be addressed in future coverage 


evaluation studies. 


KEY WORDS: Poststratification; Influence statistics; Linearization; Synthetic estimation. 


1. INTRODUCTION 


The Post Enumeration Survey (PES) of the 1990 
Decennial Census of the United States was designed to 
produce coverage estimates for 1,392 poststrata. The nation 
was first divided into 116 domains, called poststratum 
groups (PSGs) according to geography, race/Spanish 
origin and tenure (owner vs. renter). With only 4 excep- 
tions, all PSGs are defined within a census division, one 
of nine contiguous geographic areas each composed of 
several states. Each PSG was further divided into 12 age- 
by-sex groups, the poststrata. For example, roughly all 
Black renters in New York city constitute a PSG and all 
females, age 0-9, of this PSG make a poststratum (PS). 
Further details on the PES are in Hogan (1992,1993). 

Small area undercount rates were calculated by synthetic 
estimation; the same adjustment factor was applied to 
persons from a given PS in all areas. This procedure is 
accurate under the ‘‘synthetic assumption’’ of homogeneity 
of undercount rate within a PS. The validity of the syn- 
thetic assumption has been hotly debated (Section 2). This 
paper reports on research conducted as a part of a PES 
evaluation project (the ‘‘P12 project’’) which investigated 
heterogeneity within poststrata. In particular, this research 
focused on the following question: can differences in 
coverage be identified between parts of a poststratum that 
fall into different states? 

Under the homogeneity assumption, the rates are the 
same within a PS regardless of state. Thus, this assumption 
can be tested by comparing rates from state to state within 
a PS; this test focuses attention on the question of whether 
synthetic estimation is ‘‘unfair’’ to certain states. The unit 


of analysis is the intersection of a census block and a PS 
or PSG, called a block part (BP) for the analysis of the 
undercount rate data. A census block is a small area 
bounded by visible features such as streets, streams efc. 
and/or by political boundaries. In urban areas it roughly 
corresponds to a city block. In fact, most of our analyses 
are performed on PSGs, since the age-sex breakdown of 
the PSG did not vary much from state to state. Hence, the 
analysis focuses on whether BPs differ between states 
within PSG. 

Two distinct analyses were performed. The distributions 
of five ‘‘surrogate variables’’ were investigated (Section 3), 
using a large (4.26%) extract from the census. The distri- 
bution of undercount was investigated using the much 
smaller PES data set (Section 4). For more detailed tables 
and documentation of the project, see Kim (1991). 


2. LITERATURE REVIEW 


Two key questions have been addressed in the literature 
on heterogeneity: 


1. The empirical question: how much heterogeneity is 
there, and how can it be described? 


2. The theoretical and policy question: what are the impli- 
cations of heterogeneity for the accuracy of synthetic 
adjustments and the validity of assessments of these 
adjustments? 


Heterogeneity may be identified and analyzed at many 
levels of aggregation. Perfect homogeneity of undercount 
rates for very small domains is numerically impossible, 
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because of discreteness of the true population and the 
census counts. Indeed, because census errors (omissions 
or erroneous enumerations) tend to be either independent 
of each other or positively associated (as when a household 
with several members is omitted, or when some local 
characteristic affects an entire block), we would anticipate 
at least binomial variability in observed undercount rates. 

Hengartner and Speed (1993) analyzed 1990 PES data 
from two sites by fitting models in which the explanatory 
variables were block and ‘‘demoid’’ (a unit defined by 
the non-geographic poststratification variables, such as 
race, sex, age, and tenure). They found that the amount 
of variance explained by block was slightly greater than 
the amount explained by demoid; the number of blocks 
was not much greater than the number of demoids in 
their data set. In response, Schafer (1993) argued that an 
estimation scheme involving block effects would not be 
practical because it would require collecting data from 
every block. 

Heterogeneity of undercount at any level may be 
defined as excess variability in observed undercount rates 
at that level over what would be expected as a consequence 
of variability at a lower level of aggregation. For example, 
confining our attention to a single poststratum, a set of 
blocks are heterogeneous if their undercount rates in that 
poststratum differ more than would be expected if house- 
holds, including those counted, partially counted, and 
omitted in the census, had been randomly distributed 
across the blocks. Similarly, a group of states are hetero- 
geneous (similarly controlling for poststratum) if they 
differ more than would be expected if blocks, including 
those with higher and lower undercounts, had been 
randomly distributed across the states. Several studies 
have attempted to measure heterogeneity in undercount 
rates and other census variables. Wachter and Freedman 
(1992) analyzed a large sample of census data (similar to 
that considered in Section 3). They estimated the excess 
variability between ‘‘superblocks’’ over that predicted by 
a binomial model with uniform rates, for four ‘‘artificial 
population’’ variables (multi-unit housing rate, non- 
mailback rate, allocations, and substitutions, described in 
Section 3). Compared to the greatest possible amount of 
heterogeneity (if each block were homogeneous), the 
““excess variability’’ ranged from around 20% (for multi- 
unit housing) to 2% (for substitutions). Another study by 
Freedman and Wachter (1993) examined between-state 
heterogeneity using ‘‘artificial populations’’ based on the 
same variables and two others, and found substantial 
variability. 

Alho, Mulry, Wurdeman and Kim (1993) used condi- 
tional logistic regression models to describe heterogeneity 
associated with measured covariates that were not captured 
in the poststratification. Their concern was primarily with 
reducing the bias of dual system estimates of population, 
rather than with more accurate small-area estimates. 


A controversial topic in evaluation of the proposed 
adjustment of the 1990 census was the effect of hetero- 
geneity on the accuracy of adjusted population counts 
obtained by synthetic estimation, and particularly on 


‘comparisons of the accuracy of adjusted and unadjusted 


counts. Wachter and Freedman (1992) argued that because 
the ‘‘synthetic assumption’’ of uniform coverage within 
poststrata is demonstrably false, aggregate measures of the 
accuracy of an adjusted census systematically under- 
estimate error. Because nonuniformity of coverage affects 
the accuracy of an unadjusted census as well, however, the 
implications of this conclusion for the appropriateness of 
adjustment are not obvious. 

In one of the earlier ‘‘surrogate variables’’ studies, 
Isaki, Schultz, Diffendal and Huang (1988) simulated the 
behavior of synthetic estimators on “‘artificial populations”’ 
which were transformations of the substitution (unit 
imputation) rate. They found that a synthetic estimator 
generally did better than ‘‘unadjusted’’ counts. 

Schirm and Preston (1987) argued, using analytical 
calculations and simulation, that synthetic estimation makes 
estimates for small areas more accurate under plausible 
conditions, even if the synthetic assumption does not hold. 
Wolter and Causey (1991) investigated the performance 
of synthetic estimators and of a single ratio adjustment 
when the undercount rates are estimated with error, using 
undercount rates from the 1980 Post-Enumeration Program 
(PEP) and simulating various levels of sampling error; 
they estimated ‘‘break-even”’ coefficients of variation at 
which sampling error in the adjusted counts or proportions 
would make them less accurate than unadjusted counts or 
proportions. The conclusions of these articles were criticized 
by Freedman and Navidi (1992), who gave counterexamples 
of possible distributions of undercount for which adjust- 
ment by synthetic estimation would make population 
distribution less accurate. 

Fay and Thompson (1993) simulated effects of hetero- 
geneity on accuracy of synthetic estimates, using eight sur- 
rogate variables (including the five used in this study) and 
the same data set as analyzed in Section 3. They performed 
a loss function analysis as in Mulry and Spencer (1993) to 
compare the accuracy of simulated unadjusted counts to 
that of synthetically adjusted counts. They found that the 
effect of ignoring heterogeneity was to underestimate the 
gain in accuracy due to synthetic adjustment for five of 
eight variables, and to overestimate it for one variable 
(unemployment rate), while there was little difference for 
two other variables (poverty and migration rates). 


3. ANALYSIS OF SURROGATE VARIABLES 


In the analysis of census data, we selected variables 
which were available for the entire census and which, 
like undercount, were descriptive of or related to the 
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census-taking process. The selected surrogates are the 
allocation rate, mail return rate, multiunit structure rate, 
mail universe rate (fraction of units receiving mail ques- 
tionnaire) and substitution rate. The allocation rate is the 
fraction of households for which imputations were made 
for item nonresponse, and the substitution rate is the 
fraction of households which were imputed as a whole 
because it was determined that a unit was occupied but no 
interview could be obtained. 

Table 1 shows correlations between each of these variables 
and undercount rate by PSG. These ‘‘ecological’’ correla- 
tions (Freedman, Pisani and Purvis 1978, pp. 141-142) of 
PSG averages differ from those which could be calculated 
from block-level data. The latter are smaller, possibly 
because of the noise introduced by random variability in 
the small populations in each block. 


Table 1 


Correlation Coefficients between the 
Surrogate Variable 
and Undercount Rate by PSG 


Variable Correlation 
Allocation Rate .44 
Mail Return Rate — .57 
Multiunit Structure Rate 39 
Mail Universe Rate .08 
Substitution Rate .47 


Applying a naive test which treats the PSGs as indepen- 
dent, each correlation is significant except that for mail 
universe rate, but the magnitudes of the correlations are 
not large. To some extent, furthermore, these variables are 
descriptive of conditions which tend to lead to higher 
omission rates (allocations due to poor completion of 
questionnaires, substitutions due to difficulty in obtaining 
interviews) or to lower omission rates (high mail return 
rates). On the other hand, difficult census-taking condi- 
tions can also lead to erroneous enumerations, so these 
effects on net undercount are not entirely clear-cut. We 
do not analyze these variables simply because we believe 
that they are distributed in exactly the same way as under- 
count. Rather we hope that by obtaining results on the 
distributions of a range of different census variables, we 
may gain some insight into the distribution of undercount. 

For the analyses of the surrogate variables, a stratified 
cluster sample of 1990 Census data was extracted. This 
sample is composed of 204,394 blocks corresponding to 
125,000 block clusters. A block part containing less than 
ten persons was combined with successive block parts 
(in order by block number) until a minimum count of ten 
persons was obtained. This operation was performed to 
obtain relatively stable rates for the surrogate variables 
which allows us to analyze the rates themselves. 
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Surrogate variables are analyzed by logistic regression. 
Two forms of logistic regression model were used. For the 
within-PSG analysis, the model for PSG / is 


log | PL hictab ay Saag tC. 
and for the within-division analysis, 


where P;; is the rate for a surrogate variable in the i-th 
PSG and j-th state, A is the intercept, B; is the i-th PSG 
effect and C; is the j-th state effect. The models used only 
the 99 PSGs astride two or more states. Models were built 
for surrogate variables in the 99 PSGs and in each of nine 
divisions. SAS PROC CATMOD estimated the parameters 
by maximum likelihood and provided Wald statistics for 
testing the significance of state effects. 

The data were collected with a cluster sample rather 
than a simple random sample so the test statistics must be 
divided by a design effect. We estimate a design effect, 


i 


Da: Nijx (Dijk — Bij)? 
Dy, = k= - - 
Kipp - Bij) 
where fj, is the rate for the i-th PSG, j-th state and k-th 
combined BP; nj is the size of the combined BP; K;;is the 
sample number of combined BPs in the i-th PSG in the 
j-th state and fH; is the estimated rate for the i-th PSG and 
j-th state. The fraction is the ratio of the observed between- 
block variance to that expected under binomial sampling. 

The estimated design effect Dj is a measure of within- 
state within-PSG heterogeneity. The more within-state 
heterogeneity there is, the greater the sampling variance 
of the state-level rate and the harder it is to detect a signifi- 
cant difference. The magnitude of the design effect thus 
affects the statistical power of the hypothesis tests. 

The calculated design effect only approximates the 
required correction. First, Dj sums over the combined 
BPs rather than individual BPs. Second, the sample is a 
stratified cluster sample, and most or all post-strata span 
several sampling strata. The formula is only strictly correct 
for an unstratified sample. Third, the correct effect 
involves off-diagonal (covariance) as well as on-diagonal 
(variance) terms, and the off-diagonal terms are omitted. 
To account for the above, the calculated design effects 
were multiplied by the judgmentally chosen factor, 1.25. 

A design effect was calculated for each surrogate 
variable and PSG. It is small (around 2) in most PSGs for 
the allocation and substitution rate. The effect is slightly 
higher for mail return rate, but it tends to be large (as much 
as 20) for multiunit structure and mail universe rate, since 
these characteristics are usually fairly uniform within a 
block but vary greatly between blocks. 
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Table 2 summarizes the design-corrected tests for state 
effects within PSG. 


Table 2 


Number of PSGs with Significant (a = .05) 
State Effect (Logistic Regression) 


Div: ae Alloc hy aa ah Sub 
I 5 5 5 5 1(1) 3(4) 
2 12 i Ul 12 110) 12 
3 16 15 16 16 33) 1212) 
4 8 8 8 7 5(6) 5(8) 
5 10 10 10 4(4) 7(8) 
6 15 15 13 15 eh henalte 6 
7 9 8 9 9 4(4) 8(8) 
8 7 7 7 7 2(3) 6(6) 
9 17 15 14 14 5(5) 6(12) 
Sang 94 92 95 36(43)_—«74(84) 


The numbers in (_) are the number of PSGs for which test statistics 
are available when they are less than the number of groups. 


Nationally, for each surrogate variable the state effect 
is significant for at least 84% of the PSGs. (The total 
number of PSGs varies because when a PSG falls entirely 
within one state or when only one state has non-zero obser- 
vations for a particular variable, the corresponding model 
cannot be fit). The results are summarized at the division 
level. (Divisions 1 through 9 are New England, Mid- 
Atlantic, South Atlantic, East South Central, West South 
Central, East North Central, West North Central, Mountain 
and Pacific Divisions.) 

Table 3 shows the magnitude of state effects, expressed 
as x~ values of test statistics adjusted for design effect, 
for three variables having relatively high correlation with 
the undercount rate. In the table, the x* values have from 
1 to 8 degrees of freedom. 


Table 3 


Magnitude of State Effects with respect to 
Test Statistics 


Allocation Mail Return Substitution 
Rate Rate Rate 
Minimum 4.3 0.28 5.46 
25%-ile Die 102.83 49.80 
50%-ile 68.9 254.49 97.35 
75%-ile 140.3 644.05 260.88 
Maximum 945.2 8,779.88 tated) (53-1172 


In division-level models with state and PSG effects, 
both the state and PSG effects were significant at the 1% 
level in every division and for every variable (excluding 
mail universe rate in two divisions where a test statistic 
could not be calculated). 


4. ANALYSIS OF UNDERCOUNT RATE 


The results described above for surrogate variables were 
obtained early in the census process, but they have limited 


relevance to homogeneity of undercount itself. After PES 


data were processed, direct analysis of the distribution of 
undercount became possible. 

The data set for these analyses merged two data sets for 
the 12,124 PES sample blocks, one for the E-sample 
(Census follow-up) and the other for the P-sample (PES). 
There were 12,124 collection blocks, some of which were 
split up for tabulation, giving 12,964 tabulation blocks. 
More importantly, because some of the smaller blocks 
were combined in the sampling, there were 5,293 block 
clusters sampled. Correct enumerations and E-sample 
total counts are on the E-sample file. The P-sample file 
includes P-sample total counts and counts of matches 
(P-sample cases that were included in the Census). 


4.1 Variance Explained by State and PSG 


For each division, atwo-way ANOVA was fitted to under- 
count rates for state parts. Table 4 shows the ratio of the sum 
of squares due to PSGs to that due to states within a division. 


Table 4 


Variance of Undercount Rate Explained 
by State and PSG 


Div No. of No. of SS (Group) MS (Group) 
; Groups States* SS (State) MS (State) 
1 5 6 4.51 5.64 
2 12 3 4.88 .89 
3 16 9 12.69 6.77 
4 8 4 8.73 3.74 
5 10 4 Srl, piers) 
6 15 5 7.67 2.19 
| 9 7 2.78 2.09 
8 7 8 1.31 1.53 
9 7 5 40.28 10.07 
* States include D.C. 


The ratio is always greater than one and in Division 9 
it is 40.28, showing much larger effects for PSG than for 
state. The mean square for group also exceeds the mean 
square for state in each division except Division 2. This 
supports the decision to use the PS rather than the state 
as the cell for undercount estimation and adjustment. 


4.2 Tests for State Effects on Undercount Rates 


Assuming the substitution rate (fraction of units 
imputed for nonresponse) is negligible, the adjustment 
factor (R) for a domain is 


roid WCE/WE 
WM/WP’ 
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and the undercount rate is 
ly — AAR, 


where WE and WP are the estimated population sizes 
weighted up from the E and P-sample, respectively. WCE 
is the weighted number of correct enumerations and WM 
is the weighted number of matches in the PES. 

The statistic for the influence (see Appendix) of the i-th 
BP on the adjustment factor or undercount rate is 


wesc ea WCE; WP; WE; WM; 

iss WP WE al ; 
where WCE;, WP;, WE; and WM, are contributions 
from the i-th BP to the totals above. 

A linear model was fitted to BP influence statistics to 
test for state effects. Under the null hypothesis, all the state 
parts in a PSG have the same undercount rate and the 
expected mean of the influence statistics for each state is 
0 within each PSG. The influence statistics can be analyzed 
with one way ANOVA within a single PSG or two way 
ANOVA for all PSGs within a division. 

Table 5 summarizes the tests for state effects on linear- 
ized statistics within each PSG. 


Table 5 
Analysis of Linearized Undercount at the PSG Level 


Number of PSG with 


Division Number of PSG P< 05 
1 5 0 
2 1 3 
3 16 4 
4 8 5 
5 10 2 
6 15 1 
7 9 0 
8 7 1 
9 17 3 

Sum 99 19 


The tests reveal significant heterogeneity between states 
in 19 out of 99 groups at the 5% significance level. The 
magnitude of the estimated state effect ranges from a few 
percent up to 20%, but the standard errors of these esti- 
mates are very large. 

Table 6 summarizes the results of these analysis by place 
type. Place types 0, 1, 2 and 3 are large central cities in a 
Primary Metropolitan Statistical Area (PMSA), place 
types 4, 5 and 6 are non-central cities in PMSA with large 
central cities and place types 7, 8 and 9 are other areas. 

The significant results are concentrated in PSGs for 
small areas (place types 7, 8 and 9). Ten out of 32 such 
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groups show significant interstate heterogeneity at the 5% 
level. This suggests that the poststratification can be 
improved in those areas. 


Table 6 


Summary of Analysis of Linearized Undercount 
by Place Type 


Number of PSG with 


Place Type Number of PSG eee 6s 
0 11 3 
1 23 1 
2 12 1 
3 8 1 
4 0 0 
5 6 D 
6 6 1 
7 Bt 3 
8 11 4 
9 10 3 


Table 7 shows the F-statistics and p-value for state 
effect for state x PSG models, once weighted by the size 
of domain and once without weights. 


Table 7 


State Effects by Division - Weighted 
and Unweighted Data 


Unweighted Weighted 
Dion DE. Models Models 

F Dp F Dp 
1 5 57 sf 40 85 
2 2 4.64 01 72 18 
3 8 43 91 .65 74 
4 3 64 9 66 58 
5 3 66 58 1.37 US 
6 4 60 .66 24 92 
q 6 39 .88 ey 97 
8 7 62 74 .76 62 
9 4 Ve 54 .48 75 


The additive effect of state was significant in only one 
division (p = .01) inthe unweighted state x PSG model; 
when data were weighted by size of domain, the smallest 
p-value for the state effect was .18. In both cases, the most 
significant effect was observed in Division 2, in which New 
Jersey appeared to have higher undercount rate, controlling 
for PSG, than New York. Note that the most undercounted 
area in New York (New York City) had its own poststrata. 
In eight out of ten PSGs for which New Jersey and New 
York could be compared, including nonurban areas, the 
estimated undercount for New Jersey was larger than that 
for New York. Elsewhere, because the state effects in 
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different PSGs varied in magnitude and sometimes in sign, 
and because only within a minority of PSGs in any division 
were there significant state effects, there was not signifi- 
cant evidence that in the aggregate the poststratification 
was biased against certain states. 

Table 8 shows point estimates of the state effects in linear 
models for undercount rate by state part in each division, 
with effects for state and poststratum group. (Effects are 
centered at zero by division.) In effect, these are estimates 
of interstate differences after correcting for effects explained 
by the PSG composition of the different states. 


Table 8 


Estimated State Effects on Undercount within Division 
(as percent) 


Division 1 Division 4 Division 7 
Ci, =2:42 AL —2.90 IA —1.10 
ME .74 KY 1.89 KS —0.50 
MA -—0.48 NISEONOD MN -—0.01 
NH -—0.14 TN 1.03 MO —0.66 
RI 1.43 NE 1.76 
VT 0.90 ND —0.07 

SD 0.60 

Division 2 Division 5 Division 8 
NJ 4.18 AR 1.44 AZ DAY 
NY —3.91 CO 0.68 

—0.71 
PA —0.26 re vee! ID —2.24 
OK 1.58 MT —161 
TX —2.30 NVie—=0-10 

Division 3 NM BBS 
DE -—0.42 Wik 0.08 
DC D282 WY —2.84 
FL -—0.88 wes 
GA = iL 42 Division 6 Division 9 
MD 21:32 IL 0.86 AK —0.78 
NC 0.53 IN 1.12 CAs ch.G2 
SC 0.70 MI -—0.73 Hi 018 
VA -—0O.11 OH —0.88 OR —0.26 


WV 0.11 WI —0.38 WA 0.18 


The root mean square in the analysis of variance for 
state within division, averaged across all divisions, is 
1.72 percent. Recall that only in the unweighted Division 2 
analysis were the differences between states significant, it 
must be emphasized that the estimates in Table 8 do not 
represent well-measured interstate differences. The fact 
that the estimated effects are substantial in magnitude but 
are still not statistically significant tells us that the power 
of these tests to find interstate differences, given the 
sample sizes of the PES, is not as great as might be desired. 

Another approach to the power problem is to consider 
the effect of reducing the size of the census extract used 
in analysis of surrogate variables by a factor of 25, the 
ratio of the census extract to the PES sample sizes. If we 
divide by 25 each of the chi-square test statistics sum- 
marized in Table 3, then in only 27 out of 99 PSGs would 


interstate differences have been significant for allocation 
rate (compared to 94 out of 99 PSGs with the full sample). 
Similarly, significant differences would have been found 
for 53 out of 99 PSGs for mail return rate (compared to 
92 out of 99 PSGs with the full sample), and for 14 out 
of 84 for substitution rate (compared to 74 out of 84). 
Substitution rates are comparable in magnitude to under- 
count rates; after our hypothetical reduction of sample 
size, we obtain similar numbers of significant tests for 
substitution and undercount rates. It is plausible that with 
a much larger sample we would have found many more 
significant interstate differences, although one can only 
speculate on whether they would have been large enough 
to be of substantive concern. 


5. DISCUSSION 


This paper evaluates interstate heterogeneity in under- 
count rate and other census variables in the 1990 Census. 

The evaluation used 1990 Census data and 1990 PES 
data. When this research was first embarked upon, the 
PES data were unavailable and were not expected to 
become available for analysis before the scheduled com- 
pletion date. Surrogate variables from the 1990 Census 
were tested for significant heterogeneity among states 
within PSG. At the PSG level, state effect was significant 
(a = .05) for 84%-95% of its PSGs for the various 
surrogate variables. 

ANOVA on linearized undercount based on the PES 
data at the PSG level showed significant (a = .05) state 
effects for 19 out of 99 PSGs. The significant results were 
concentrated in the PSGs in non-PMSA areas. Ten out of 
32 such PSGs had significant state effects. This suggests 
that the poststratification in the relatively nonurban areas 
was not as successful as in the more urbanized areas. 

How can we explain the different findings of the two 
studies? The two data sets had very different sample sizes, 
i.e., the Census data had 125,000 block clusters but the 
PES data had 5,293 block clusters. It is therefore not 
surprising that small differences between states on surrogate 
variables would be statistically significant although 
corresponding differences would not be demonstrable with 
respect to undercount rates. 

Furthermore, the correlations between the undercount 
rate and the surrogate variables are low as shown in 
Table 1. Therefore, any generalization from surrogate 
variables to undercount rates is somewhat conjectural. 
Given the modest correlation between undercount rates 
and surrogate variables, we prefer to give greater weight 
to the analysis of the PES data. 

We conclude from these data that there are no demon- 
strable differences in average undercount rate between 
states within each division, after adjusting for PSG effects. 
While there is weak evidence for a difference between 
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New Jersey and New York within the Mid-Atlantic division, 
this result must be downweighted in the context of the 
number of divisions (nine) for which the test was performed. 
We conclude that if adjustment of population counts had 
been carried out based on the 1990 PES, no state would 
have been able to show that the poststratification was 
manifestly unfair in that it underadjusted that state relative 
to what direct state estimates showed that it deserved. 

As the review in Section 2 shows, there is no consensus 
on whether or not between-state heterogeneity in under- 
count rates within PSG which is of substantial magnitude, 
although not large enough to be accurately measured by 
PES, would systematically affect the gain in accuracy 
obtained by synthetic adjustment. Nonetheless, the differ- 
ences between states that were identified in analysis of the 
PES, together with the ancillary evidence of the surrogate 
variable analyses, make it appear likely that heterogeneity 
between states will again be an issue in coverage measure- 
ment for the year 2000 census, especially for the larger states 
for which these coverage differences can be most accurately 
measured. Fay and Thompson (1993) argue that a coverage 
measurement sample for 2000 should be designed to 
support direct (rather than synthetic) estimates of under- 
count for all states, although a CNSTAT panel (CNSTAT 
1994) warns that for some states this could impose a highly 
inefficient sample allocation. Research over the intervening 
years must address the development of a combination of 
sample design and estimation methods that will produce 
defensible estimates of population by state. 
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APPENDIX 


Testing for Interstate Differences Using 
Linearized Statistics 


A two-way ANOVA for adjustment factors in state 
parts yields an intuitively meaningful summary of the 
relative contributions of state and PSG effects to the 
variation in adjustment factors. Because the sampling unit 
of the PES is the block cluster rather than the state part, 
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these models do not yield valid statistical tests of the 
significance of the state effects. 

Consider a statistic whose sample estimate for a state 
or state part is a weighted mean of the sample estimates 
in each component block or BP. Significance of the state 
effects for this statistic within a PSG could be evaluated 
by one-way ANOVA with the included block parts as units 
(corresponding to PSUs), or aggregated across PSGs by 
two-way ANOVA for state and PSG effects. 

The sample adjustment factor estimate (WCE/WE)/ 
(WM/WP) is a nonlinear function of sample counts. In 
small primary sampling units (PSUs) such as block parts 
this nonlinearity may be very noticeable, especially when 
the number of matches in a PSU is very small or zero so 
that the sample estimate of the adjustment factor is large 
or infinite. In this situation, if PSU sample estimates are 
treated as data, the additive assumptions of ANOVA are 
violated. Useful tests may be recovered, however, by using 
a linearized version of the statistic of interest. 

Suppose that we are interested in a parameter Z = f(X) 
where X is a vector of population proportions in certain 
cells. Let X, x; represent the corresponding sample pro- 
portions in the entire sample and in PSU / respectively, so 
X¥ = YN;x;/¥N;is a size-weighted average of block cell 
proportions. Let f; (X) be the gradient of fat _X. Then by 
Taylor linearization f(x) — f(X) =~ f,(X)’ (% — X) = 
YN fi (X)'xX;)/¥N; — fi (X)'X, i-e., we may treat the 
problem as one of inference regarding the quantities 
(pseudo-observations) z; = f; (X)’x;. Because the approx- 
imate (linearized) influence of PSU /on the estimate f(x), 
that is, the difference between the estimate with and 
without PSU i included, is N;f,(X)’ (x; — x), we may 
describe this as a method based on influence statistics 
(Hampel et a/. 1986) or the infinitesimal jackknife (Efron 
1982, Chapter 6). 

To derive a sensible variance model, suppose that we 
may regard PSU ij as sample (not necessarily SRS) from 
a superpopulation with cell proportions X;. A simple 
model is then, for some covariance matrices U; and V;, 


superpopulation model: 
ECX) SEX Var(X)e= V; 
and 
sampling model: 
E(x; | X;) = Xj, Var(x; | Xj) = Uj. 


The sampling covariance U; will typically be propor- 
tional to N;_'. A plausible and mathematically conve- 
nient specification for V;is V; w N;' (i.e., smaller PSUs 
more variable than larger ones), so Varz; = o7/N; for 
some constant 0”. The corresponding linear model weight 
for PSU i is N; so the model-based estimate of the mean 
agrees with the design-based estimate obtained by aggre- 
gating the cell counts if N; is a weighted size measure. 
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In the case of the adjustment factor R = (WCE/WE)/ 
(WM/WP), the pseudo-observations are of the form 
Zi = fi (X)' (% — X%) = 


2 WCE; re WP, WE, WM, 
WCE WP WE WM )’ 


where WCE;, WP;, WE; and WM, are similar to the 
above for the i-th BP. We approximate the appropriate 
weight of a block part by N; = (WE; + WP;)/2. 

If the variance specifications of the model are inaccurate 
so there is some heteroscedasticity, or if the distribution 
is very long-tailed, then there will be a long-tailed distri- 
bution of residuals, making the tests at least slightly 
liberal. Some care must be taken to note the presence of 
outliers signaling this heteroscedasticity, for example, 
outlying blocks due to large-scale geocoding errors. 

The assumption of approximately independent obser- 
vations in ANOVA may be violated in two ways. First, the 
PSUs are not selected by SRS but rather by a geographical 
stratification somewhat finer than reflected in the post- 
stratification scheme. To the extent that this geographical 
stratification reduces the sampling variance of the state 
effect estimates, inferences under the independence model 
will be somewhat conservative. Second, there will be 
correlations between adjustment factors for different 
block parts from the same block (in multi-PSG models). 
These will tend to make inferences assuming independence 
somewhat liberal. On the balance, we regard the tests 
performed here as useful. 
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Markov Chain Designs for One-Per-Stratum Sampling 


F. JAY BREIDT! 


ABSTRACT 


Classical results in finite population sampling tell us that systematic sampling is the most efficient equal-probability 
one-per-stratum design for certain kinds of autocorrelated superpopulations, but stratified simple random sampling 
may be much better than systematic sampling if the superpopulation is a trend with uncorrelated errors. What if 
the superpopulation consists of a trend plus autocorrelated errors? Intuitively, some sort of ‘‘compromise’’ between 
the two designs might be better than either. Such compromise designs are constructed in this paper and are shown 
to be examples of Markov chain designs, a wide class of methods for one-per-stratum selection from a finite 
population. These designs include as special cases systematic sampling, balanced systematic sampling and stratified 
simple random sampling with one sampling unit per stratum. First and second-order inclusion probabilities are 
derived for Markov chain designs, yielding the Horvitz-Thompson estimator and its variance. Efficiency of the 
Horvitz-Thompson estimator is evaluated using superpopulation models. Numerical examples show that new designs 
considered here can be more efficient than standard designs for superpopulations consisting of trend plus auto- 
correlated errors. An example of the implementation of Markov chain designs for the 1992 National Resources 


Inventory in Alaska is given. 


KEY WORDS: Balanced systematic sampling; National Resources Inventory; Systematic sampling. 


1. INTRODUCTION 


A stratified sampling design, in which a finite population 
is divided into non-overlapping strata and samples are 
drawn from each stratum, is a common and effective 
technique for reducing sampling error. In practice, strat- 
ified sampling designs with only one sampling unit per 
stratum are widely used. Examples include stratified 
simple random sampling and systematic sampling with its 
variants (e.g., Murthy and Rao 1988). 

Systematic samples are susceptible to systematic errors. 
In large-scale spatial samples, for example, sources of 
systematic error could include roads, powerlines, irrigation 
systems, and so forth. A favorite example is the system of 
‘*section roads’”’ in areas of the United States covered by 
the public land survey. This grid-based system is built up 
from square tracts of land called sections, each one mile 
on aside, which are often bounded by roads in midwestern 
agricultural regions. A systematic sampler with a one-mile 
sampling interval and an unlucky random start might 
conclude that Iowa is covered by gravel roads! 

Systematic sampling does have the advantage of effi- 
ciency when the sampled population is positively auto- 
correlated, as is often the case in temporal and spatial 
sampling problems, since it forces observations to be as 
distant and hence as uncorrelated as possible. 

Both autocorrelation and systematic error are of concern 
in the National Resources Inventory (NRIJ), an area sample 
of the nonfederal lands in the United States conducted 
every five years by the Soil Conservation Service of the 


United States Department of Agriculture. NRI data items, 
collected by a combination of remote sensing and ground 
observation, include soil characteristics, land use, agricul- 
tural practices, erosion measures, and so on. 

The 1992 NRI sample design for the northwestern 
region of the state of Alaska is a controlled version of 
one-per-stratum sampling. The region was divided into 
twenty-minute bands of latitude. Each band was divided 
into 500,000-acre strata. Each stratum was divided into 
al0 x 10 grid of cells indexed by latitude and longitude, 
and one cell per stratum was selected. Selection moved 
from east to west across the strata within a particular 
twenty-minute band. The random numbers which deter- 
mined the longitude cells of the selected units and the 
random numbers which determined the latitude cells 
evolved as two independent Markov chains. (Basic results 
on Markov chains used in this paper can be found in an 
introductory text on stochastic processes such as Taylor and 
Karlin 1984). Details of the design are given in Section 2. 

How does this ad hoc design compare to more standard 
one-per-stratum designs? It turns out, as shown in Section 2, 
that simple Markov chain techniques can describe a broad 
class of equal-probability designs for one-per-stratum 
selection from a finite population. This class includes 
standard techniques such as stratified simple random 
sampling, systematic sampling and balanced systematic 
sampling, as well as the Alaska designs described above. 
It is also easy to generate new designs within this class. This 
unified treatment of one-per-stratum designs allows for 
comparisons of efficiency. 
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First and second-order inclusion probabilities for all of 
these designs are derived in Section 3, yielding the Horvitz- 
Thompson estimator and its variance. As in much of the 
relevant literature (Madow and Madow 1944; Cochran 
1946; Sedransk 1969; Bellhouse and Rao 1975; Wolter 
1985; Bellhouse 1988; efc.) the average design variance of 
the Horvitz-Thompson estimator is evaluated under a 
variety of superpopulation models. Compact expressions 
for model-averaged design variances are obtained. Numer- 
ical examples in Section 4 show that designs introduced 
in this paper can be more efficient than standard one-per- 
stratum designs for superpopulations consisting of trend 
plus autocorrelated errors. Discussion follows in Section 5. 

Though our motivating example is two-dimensional, 
one-dimensional designs will be considered throughout. 
Most proofs and derivations are straightforward and are 
omitted for brevity. 


2. MARKOV CHAIN DESIGNS 


Consider the problem of sampling from a finite popula- 
tion of N = na labeled units, denoted by 


OU -= vil, ee iN} 


ae 1S A ONG “tral uaa 2 Ceara 


(7 — 1a + I, ....,. nay. 


The value of a study variable yy = Y(j-1)a+j; = Vy 18 
associated with each label k; the notation y, or yj; will be 
used for both random variables and realizations of 
random variables. 

Here n is the sample size and a is the sampling interval. 
The n subsets 
Pra" a te ee (Dae apne = Wen) 
will be referred to as strata. The goal is to select one unit 
per stratum. Often, a stratified sampling design is defined 
to be one in which independent probability samples are 
selected in each stratum, but the restriction to independence 
is not used here. 

Given a doubly stochastic transition probability matrix P, 
a Markov chain sample is given by 

Si=SetR a HRs aa) ja GOR, 
where R,, ..., R, is the Markov chain defined by P and 
R, ~ uniform (1, ..., a). Formally, then, a Markov 
chain design (MC) is a function p( * ;P) such that 


D(s;sP) = Pr{s = {7,a +n, ..., (1 -— lhat+nr}} 


Pr{R, = TN}, R, = ly bas 
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P 


els exis ‘ 


Pp 


+1579 @> 


= FOLD SHEP TIE (lca iQ). 


0, otherwise. 


MC designs as defined in this paper are related to the 
designs given in Chandra, Sampath and Balasubramani 
(1992), in whichal x Nvector of initial selection proba- 
bilities and a N xX N transition probability matrix of 
periodicity n determine a without-replacement sampling 
scheme. Chandra e¢ al. focus on producing designs with 
strictly positive second-order inclusion probabilities. They 
do not explicitly consider the one-per-stratum designs of 
this paper, which can be imbedded in their structure in a 
straightforward way by constructing the appropriate initial 
probability vector and transition probability matrix. 

The following result is useful in deriving the probabilistic 
features of MC designs. 


Result 1 Consider a Markov chain for which the tran- 
sition probability matrix P is doubly stochastic (i.e., all 
row sums and all column sums equal one) and R;, has a 
discrete uniform distribution, with mass 1/a on each of 
the states 1, ..., a. Then R; has a discrete uniform distri- 
bution on the states 1, ..., a for all 7. In particular, R; has 
mean (a + 1)/2 and variance V(R;) = bani: 


Some special cases of MC designs are of interest. 


Stratified simple random sampling. If the transition 
probability matrix is 


H = [1/a]j,j-=1> 


Pr{R; = fj’ | Rj = J} = l/a = Pr{R) = /'} 
© OR aaa penetrate 8 


which, together with the Markov property, implies that 
R,, ..., R, are probabilistically independent. In this 
case, the MC design is stratified simple random sampling 
with one unit per stratum (ST). 


Systematic sampling. If the transition probability 
matrix is J, the a X a identity matrix, then 


i. -; 1 j — los 
Pr{Ry =j' | Ri =) = { i 1 
OS Jiri dibs 
so that R,, ..., R, are deterministically related. Thus, 
S= {R),a + R, ...,(n — la + Rj}, 


and so the MC design is systematic sampling (SY). 
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Compromise designs. Intuitively, ST and SY are at 
opposite ‘‘extremes’’ in some sense. If p € [0,1], then 


Gi= pH + (1 = p)t 


is doubly stochastic. If p = 0, the design is SY and if 
p = 1, the design is ST. Any other choice of p will yield 
a sequence consisting of ‘‘runs’’ of SY samples. Thus, the 
class G, includes ST and SY, as well as a continuum of 
“‘compromise’’ MC designs. 

Other convex combinations of doubly stochastic 
matrices could be considered. The class of doubly stochastic 
matrices is also closed under matrix multiplication, trans- 
position, and row and column permutation, so there are 
many ways to create MC designs. 


Balanced systematic sampling. Murthy (1967, §5.9d) 
describes a one-per-stratum selection method which he 
calls balanced systematic sampling (BA). This method 
gives samples 


Ss = {R;,a + (a + 1—R)), ..., (n — 2)a + Rj, 


(n — l)a + (a + 1 — R,)} 
for n even and 


s= {R,,a+ (a+ 1-—R)),...,(n — 2)a t+ 


(a;+ 1,.— Ry) (ni— 1)a: 43R)4 


for n odd. An interesting feature of this design is that 
if n is even and the population is perfectly linear (yj; = 
Bo + B,[(@ — 1)a + /]), then the sample mean equals 
the population mean for any sample. With the transition 
probability matrix, 


OHO. 770i 
OO Esaar kas 

wi , 
0,1 s. <0 0 


1 ORO sO RB aexXea 
BA is a MC design. 


Alaska NRI design. As described in Section 1, the 1992 
NRI sample design for the northwestern region of the state 
of Alaska used two independent Markov chains in the 
controlled selection of latitude and longitude cells. The tran- 
sition probability matrix for longitude cells, Pong, is given 
in Table 1. This design, henceforth denoted AK, is a MC 
design since Pong is doubly stochastic. Most of the tran- 
sition probabilities are close to 0.10, so most ‘‘step sizes”’ 


65 


are approximately equally likely. Note, however, that mass 
has been removed from on and near the back diagonal 
and placed in the upper left and lower right corners, so 
that Pong discourages large east to west steps, such as 
from cell one to cell ten, and discourages small steps, such 
as from cell ten to cell one. On the other hand, Pong 
encourages steps of around length ten, such as from cell 
two to cell one, two or three. The realized sample of 
longitude cells is thus well-dispersed east to west, like a 
systematic sample would be, but its additional randomness 
guards against systematic error. Similarly, the Markov 
chain for latitude cells was set up to give good spatial 
dispersion north to south. 


Table 1 


Transition probability matrix for Markov chain sample 
of longitude cells, 1992 National Resources Inventory, Alaska. 
Entries are the conditional probabilities of selecting 
cell’ of stratum/ + 1 given that cell / of stratum i was selected. 


Cell j of Cell j’ of stratum i + 1 


stratumi =; Sep MONE ee i ak Spe i aS A elk Ra (3) 
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3. HORVITZ-THOMPSON ESTIMATION 
UNDER MC 


Write the population total as 
n a n a 
las Drove Fr yy amnesty = De i dye 
U i=1 j=l i=1 j=l 


For all k, the first-order inclusion probabilities of a MC 
design are given by 


a, = Pr{k €s} = Pr{R; = Jj} = l/a 


and for k < /, the second-order inclusion probabilities are 
given by 


~ 

~,. 
a 

K 
~ 


Th = 0, for 
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The design-unbiased Horvitz-Thompson estimator 
(Horvitz and Thompson 1952) for the population total 
is then 


The design covariances of the indicators /;r,-;, are given 
by 


Cuc(liraay Ler 253) = Euc[iaaa ley =i) — 
Exc|!(r;=1)| Emc[tR; =1')] 


S GSA SYA ae 


W(i-1)at+j ™(i'-1)atj’» 


and so the design variance of f, is 


Vuc(t,) =a? yy a ( is 2) 2090 (1) 


Since the design variance depends on all the values of 
the study variable in the finite population, (1) is not easily 
used for comparing designs. Following Cochran (1946), 
assume that the values of the study variable are generated 
from the superpopulation model 


Ei yy = by + ey, 


where the p,; are fixed and the e;; are random variables with 
E,lej] = 0, V;(ej) = of and Cy (ej, ej) = oy, 77" 
Then designs can be compared on the basis of model- 
averaged design variance. 


Proposition 1 Under the superpopulation model &, 
the average design variance of the Horvitz-Thompson 
estimator is 
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E;(Vuc(4,)] = a sD Hn # 


for any MC design. Note that if yj; is independent of /, 
then Vac[ LY 7=1 Hir;] = 0. 

The following proposition gives a sufficient condition 
under which no MC design has worse average design 
variance than SY. 


Proposition 2 Consider an uncorrelated additive model, 
fo Vy iy ey oO ta te cays 


where E;[e;;] = 0, V;(e;) 
Then 


= 07, and Ciler ei";’) = (0). 


E;(Vsy(@,)] = Esl Vuc(t)] 


for all MC designs. 


Proof From Proposition 1, the only term of interest is 
Vuc| Z je Hir,|> which under SY is 


Yer ys a =: Ver yet nba = n’V(Br,), 


i=1 i=1 


while under a general MC design, 


n n n 

ruc ye Hn =) DY Cuc(Bp;8p;-)- 

i=l i=1 i’=1 

Since Cuic(Bp;»8r;-) = V(Br,)> the proposition follows. O 
Some specific models are considered in the next five 

subsections. 

3.1 Random Permutation Model 


A model for a population in random order is a permuta- 
tion model, in which a realization of the measurements 
Yi, -.-, Yn is given by one of the N! equally likely permu- 
tations of N fixed values. This model can be written as 


Step = Sores 


where Jy = Yuy,/N. See Rao (1975) for more details. 
The following result is then a consequence of Theorem 2.1 
of Rao and Bellhouse (1978). 
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Result 2. Under the random permutation model, 
E;, (Vuc(t)] = 
Nem te UN) Ve Vee Vu) ool) 
U 


for any MC design. 


Thus, the average variance over all permutations is 
exactly Vg)(¢,), where SI denotes (unstratified) simple 
random sampling without replacement. For SY, this result 
is originally due to Madow and Madow (1944). See also 
Sedransk (1969). 


3.2 Stratification Effects Model 
A model for a population with stratification effects is 
£2: Vig = a + ei, 
where the q; are fixed constants and e,; are uncorrelated 
random variables with mean zero and variance o”. Note 


that if a; = p, then €, is an alternative to €, as a model 
for a population in random order. 


Result 3. Under the stratification effects model, 
Ex, Vacl&)] = na(a — 1)0? 
for any MC design. 


3.3 Linear Trend Model 


A model for a population with a linear trend is 
Cay iy Hebel Ui Widod alts rey; 


where @ and £, are fixed constants and e;; are uncor- 
related (0,07) random variables. 


Result 4 Under the linear trend model &;3, 
n 

Ez,(Vac(&)] = fa*¥nc| Y R| + na(a — 1)0? (2) 
i=! 


for any MC design. Since 3 is additive, no MC design has a 
larger expected variance under a linear trend model than SY. 

The only design-dependent term in (2) is Vuc| v7,R il- 
Under SY, ¥ 7_,R; = nR,, so that 


Vex dB: x = n’V(Rj), 
i=1 


while under ST, 
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Ver] yy x = nV(R)). 
ia 


Under BA, for 7 even, 


2 n n 
Yes| ¥ R = Yon aR + 2 (a +1 —-— R)| ="(0), 
This implies that if the population is perfectly linear 
(o* = 0), then 
E;,(Vea(t,)] = 0, 


so that 7, = ¢ for all samples, as noted by Murthy (1967, 
p. 165). 


Result 5 Under the linear trend model £3, 
Ez,(Vea(t,)] < Ex, [Ver (4) 
= F;,1Ve, (7) 
< E;,( Vey (¢,)] = maxE;,[Vic(4,)], 
MC (3) 
where the middle term is monotone increasing with 


decreasing p € [0,1]. If mis even, the left-hand side of (3) 
equals minycE;,[Vuc(t) 1]. 


3.4 Periodic Population Model 


A simple model for a population showing a deter- 
ministic periodicity with period p is the sine wave model 


2 
Eg: yy = asin FG = 1) val Rey. 
Dp 


where ej; are uncorrelated random variables with mean 
zero and variance o. 


Result 6 Under the periodic population model &,, 


E:,(Yuc(4)] = aarKuc| Ye sin"[(¢ - 1a + Ra 
P 


i=1 


+ nga — 1)" 


for any MC design. 


Denote the sine wave model £4 with p = a by &4,. 
Under &,,, 


2 re 
sin - l)a +a} Beinees 
D a 
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so that the model is additive and no MC design has larger 
expected design variance under &,, than SY, highlighting 
the fact that SY is inappropriate for a population containing 
a periodicity with period equal to the sampling interval 
(Madow and Madow 1944). This result generalizes as follows. 


Result 7 If y;; = 6; in €, then & is a model for a popula- 
tion showing a deterministic periodicity with period equal 
to the sampling interval, a. The model & is additive and 
so no MC design has larger expected design variance under 
£ than SY. 


3.5 Autocorrelated Model 


Beginning with Cochran (1946), many authors have 
compared ST, SY and simple random sampling under an 
autocorrelated superpopulation model. See Bellhouse 
(1988, §4) for a review. 

Consider the following autocorrelation model due to 
Cochran (1946): 


in el a ZY 
whete a; 747 — yl =e stay “Stor ee. 
Result 8 Under the autocorrelated model é;, 


a—\ 
Exs[Vaic(te)] = na(a —1)y(0) — 2n Y) y(h)(a —A) 


h=1 


+0 3 3 y(ha + j’ Jin ~ my (PAP — “| 


(jell Pesil PES 


for any MC design. 


Result 9 If, for h = 0, y(h) is non-negative, non- 
increasing and convex, i.e., 


7 (kh) Say) = 


y(h + 2) — 2y(h +1) + y(h) = 0 


y(h + 1) and 


then E;.[ Vey (4,)] = minyc£:,[ Vuc(t,)1]- 


This result is a corollary of a theorem due to Hajek 
(1959), given as Theorem 4.1 of Bellhouse (1988); Bellhouse 
clarified the conditions under which the theorem holds. 
Hajek’s theorem generalized an earlier result due to 
Cochran (1946), who compared SY, ST and simple 
random sampling. 


4. EFFICIENCY: SOME NUMERICAL 
EXAMPLES 


An important class of models for time series and spatial 
processes consists of a low-order polynomial trend plus 
an autocorrelated error sequence. A simple example is 
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E (8,6)? ¥y = Bo + Bil(Gi — 1a t+ Jf) + ey, 


where the autocorrelation structure is that of a first-order 
autoregressive (AR) model, 


Oe = YU -—Dat+s’ -Jfl = o2g i Dati’ —j 


for i’ > iand |¢@| < 1. The average design variance 
under this model is obtained from Results 4 and 8. For 
different choices of 8, and ¢, the ratio of expected design 
variances, 


VA Acret AN DEI sed ws Ihe (4) 


is given in Table 2 for various MC designs. Also tabled is 
the optimal G, design, obtained by minimizing (4) with 
respect to p. Use of this design is only feasible if super- 
population parameters are known, so it is tabled merely 
as a benchmark and not as a competitor. 

When £6, # 0 and ¢ = O, the model is £3 and the 
tabled values agree with Result 5: SY is the worst MC 
design and BA is the best, with G:,, Gz, and ST falling 
between them. Though BA does extremely well for this 
model, any non-SY MC design would be a good choice. 

When 6; = 0 and ¢ # 0, &(g4) is a special case of 
model é;. For ¢@ > 0, Result 9 and the table agree that SY 
is most efficient since it makes the sample as ‘‘spread out’’ 
as possible, but for weak autocorrelation, the other MC 
designs are competitive. BA is very poor for this model, 
because the design ensures that every other pair R;, R;+| 
will be no more than a units apart. (For the same reason, 
BA is good for a negatively autocorrelated population.) 
AK, Gy, and Gx, outperform ST, because each of these 
designs encourages state transitions of around length a. 

Similar results are obtained for the superpopulation 
model 
E(a,6) > Vij = asin ul eer 
where oj; ;/;, is as above. Table 2 gives the ratio of expected 
design variances (4) under this model, obtained from 
Results 6 and 8. 

Whena # Oand¢ = 0, the model is £4, and SY per- 
forms badly, as indicated by Result 7. Even ford ~ 0,SY 
performs well only when the periodicity is swamped by 
highly-correlated noise. 

Note that no design dominates Table 2: each of SY, Gy, 
Gy,, ST, BA and AK is the best at least once among those 
considered. For a moderate trend and high autocorrelation, 
AK, G,, and Gz, can beat standard MC designs. Overall, 
Table 2 suggests that some non-standard MC designs, such 
as Gz, and AK, do reasonably well for a variety of popula- 
tions: retaining much of the efficiency of SY against an 
autocorrelated population, while still guarding against 
systematic effects in other kinds of populations. 
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Table 2 


Ratio of expected design variance under MC to expected 
design variance under SY for superpopulation consisting of trend 
(line with slope 6, or sine wave with period a and amplitude a) 

plus autoregressive (AR) errors (N = 1,000, 0% = 100, 

a = 10). Here G,. is the optimal compromise design, 
where p* is a function of superpopulation parameters. 
Ratio for the best realizable design in each row 
(if not SY) is italicized. 


Markov Chain Design 


Oo) Gy, Gy, ST BA AK Gp* (p*) 


—0.5 | 0.2322 0.2085 0.2001 0.1666 0.2056 0.2001 (1.0000) 
TNC AR: 0.0} 0.2220 0.1983 0.1903 0.182] 0.1957 0.1903 (1.0000) 
0.1 | 0.2187 0.1950 0.1871 0.1825 0.1921 0.1871 (1.0000) 


Baa 0.5} 0.1922 0.1702 0.1645 0.1754 0.1659 0.1645 (1.0000) 
0.9| 0.0980 0.0778 0.0742 0.0768 0.0762 0.0742 (1.0000) 

—0.5] 0.4504 0.4328 0.4262 0.3647 0.4304 0.4262 (1.0000) 

TineeeAR 0.0| 0.4344 0.4172 0.4114 0.4054 0.4153 0.4114 (1.0000) 
8, = 0.4 0.1} 0.4291 0.4121 0.4065 0.4085 0.4094 0.4065 (1.0000) 
0.5 | 0.3853 0.3727 0.3724 0.4116 0.3667 0.3719 (0.8320) 

0.9| 0.1876 0.1835 0.1914 0.2170 0.1848 0.1821 (0.5223) 

—0.5 | 0.9233 0.9190 0.9163 0.7941 0.9175 0.9163 (1.0000) 

TineewAR 0.0} 0.9201 0.9177 0.9169 0.9160 0.9174 0.9169 (1.0000) 
8B) = 0.1 0.1} 0.9191 0.9175 0.9175 0.9349 0.9156 0.9174 (0.8156) 


0.5] 0.9160 0.9289 0.9439 1.0606 0.9185 0.9135 (0.1997) 
0.9| 0.8621 0.9787 1.0725 1.2710 1.0017 0.7888 (0.0981) 


—0.5} 0.9978 0.9956 0.9935 0.8617 0.9942 0.9935 (1.0000 

0.0} 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 (---) 

Pure AR 0.1] 1.0009 1.0019 1.0028 1.0228 1.0001 1.0000 (0.0000) 
0.5} 1.0179 1.0357 1.0536 1.1852 1.0245 1.0000 (0.0000) 

0.9] 1.2517 1.4380 1.5814 1.8798 1.4734 1.0000 (0.0000) 


—0.5} 0.9929 0.9906 0.9884 0.8578 0.9892 0.9884 (1.0000) 
0.0| 0.9947 0.9946 0.9945 0.9950 0.9946 0.9945 (1.0000) 


Hone ae 0.1| 0.9955 0.9963 0.9972 1.0175 0.9945 0.9954 (0.1925) 
0.5] 1.0110 1.0285 1.0462 1.1775 1.0173 0.9977 (0.0364) 
0.9| 1.2178 1.3980 1.5371 1.8294 1.4322 0.9999 (0.0018) 
—0.5| 0.6747 0.6634 0.6586 0.6008 0.6604 0.6586 (1.0000) 
Sine + AR 0.0 | 0.6603 0.6499 0.6464 0.6770 0.6477 0.6464 (1.0000) 
set ys 0.1| 0.6554 0.6455 0.6425 0.6863 0.6421 0.6425 (1.0000) 
0.5| 0.6149 0.6133 0.6196 0.7320 0.6041 0.6121 (0.5079) 
0.9| 0.3570 0.3832 0.4126 0.5527 0.3877 0.3560 (0.2852) 
~0.5] 0.0668 0.0384 0.0287 0.1101 0.0323 0.0287 (1.0000) 
Sine + AR 0-0 0.0656 0.0372 0.0275 0.1115 0.0311 0.0275 (1.0000) 
y= 10.0 0-1! | 0.0652 0.0368 0.0277 0.1115 0.0307 0.0271 (1.0000) 


0.5 | 0.0622 0.0339 0.0245 0.1106 0.0277 0.0245 (1.0000) 
0.9} 0.0529 0.0247 0.0154 0.1016 0.0187 0.0154 (1.0000) 


5. DISCUSSION 


The class of Markov chain designs has been defined and 
shown to include systematic sampling, stratified simple 
random sampling and balanced systematic sampling as 
special cases. Some new designs have been introduced 
(G,, AK) and shown to be competitive with standard 
one-per-stratum designs under a variety of superpopulation 
models. In particular, the new designs work well in numer- 
ical examples for trending superpopulations with auto- 
correlated errors. This is the kind of population of concern 
in many area sampling problems, such as the 1992 National 
Resources Inventory in Alaska. A two-dimensional MC 
design implemented for that survey shows that one- 
dimensional MC designs might be usefully extended to a 
spatial sampling context, though further work on this 
extension is necessary. 
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Further work on variance estimation for MC designs 
is also needed. Because these are one-per-stratum designs, 
design-unbiased estimation of the variance of the Horvitz- 
Thompson estimator is not possible. The problem of 
variance estimation for one-per-stratum designs, partic- 
ularly for SY, has received much attention. For example, 
Wolter (1985) discusses in detail eight different biased 
variance estimators for SY and evaluates their biases under 
superpopulation models. Work in this direction for the 
collapsed strata variance estimator (e.g., Cochran 1977, 
p. 139) under general MC designs is in progress. 
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Median Estimation Using Auxiliary Information 


GLEN MEEDEN! 


ABSTRACT 


The problem of estimating the median of a finite population when an auxiliary variable is present is considered. 
Point and interval estimators based on a non-informative Bayesian approach are proposed. The point estimator 
is compared to other possible estimators and is seen to perform well in a variety of situations. 


KEY WORDS: Sample survey; Estimation; Median; Auxiliary variable; Quantile; Non-informative Bayes. 


1. INTRODUCTION 


The problem of estimating a population mean in the 
presence of an auxiliary variable has been widely discussed 
in the finite population sampling literature. The ratio 
estimator has often been used in such situations. For 
the problem of estimating a population median the situa- 
tion is quite different. Only recently has this problem 
been discussed. Chambers and Dunstan (1986) proposed 
a method for estimating the population distribution 
function and the associated quantiles. They assumed that 
the value of the auxiliary variable was known for every 
unit in the population and their estimator came from a 
model-based approach. Rao et al. (1990) proposed ratio 
and difference estimators for the median using a design- 
based approach. Kuk and Mak (1989) proposed two other 
estimators for the population median. To use the Kuk 
and Mak estimators one only needs to know the values 
of the auxiliary variable for the units in the sample and 
its median for the whole population. The efficiencies of 
these estimators depend directly on the probability of ‘con- 
cordance’ rather than on the validity of an assumption of 
linearity between the variable of interest and the auxiliary 
variable. 

Recently Meeden and Vardeman (1991) discussed a 
non-informative Bayesian approach to finite population 
sampling. This new approach uses the ‘Polya posterior’ 
as a predictive distribution for the unobserved members 
of the population once the sample has been observed. 
Often it yields point and interval estimates that are very 
similar to those of standard frequentist theory. Moreover 
it can be easy to implement in problems that are difficult 
for standard theory. In this note we show how this method 
can be used for the problem of estimating a population 
median when an auxiliary variable is present and compare 
it to some of the other proposed methods. 


2. ESTIMATING THE MEDIAN 


Consider a finite population containing N units. For the 
unit with label i let y; denote the characteristic of interest 
and x; the auxiliary variable. We assume that both y; and 
x; are real numbers and each is known for every unit in the 
population. Let s denote a typical sample of size n which 
was chosen by simple random sampling without replace- 
ment. We assume simple random sample for convenience, 
since in many problems of this type the sampling will often 
be more purposeful. Before considering the problem of 
estimating the median of the population we review some 
well known facts about the problem of estimating the mean. 

Consider the super population model where it is 
assumed that for each /, y; = bx; + u;e;. Here b is an 
unknown parameter while the u;’s are known constants 
and the e;’s are independent identically distributed 
random variables with zero expectations. Since the popu- 
lation mean can be written as es lies ¥i + Lies¥j) We 
would expect N~!( Dies; + bY jesXj) to be a sensible 
estimate of the mean whenever b is a sensible estimate of 
b. One particular choice of b is the weighted least squares 
estimator where the weights are determined by the u;’s. 
For example if for all 7, u; = is the resulting estimator 
is just the usual ratio estimator. While if for all 7, u; = x;, 
then b = n7! Vics (y;/x;) and the resulting estimator is 
one that was discussed by Basu (1971). (See also Royall 
(1970).) Using this super population setup it is easy to 
generate populations where the ratio estimator has smaller 
mean squared error than the Basu estimator and vice versa. 
A somewhat limited simulation study on a variety of 
populations found that the performance of the Basu 
estimator is quite similar to the performance of the ratio 
estimator although in the majority of the cases the ratio 
estimator performs better than the Basu estimator. This 
is not unexpected, given the wide use of the ratio estimator. 
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In Meeden and Vardeman (1991) a non-informative 
Bayesian approach to finite population sampling, based 
on the Polya posterior, was developed. For the simple 
problem where no auxiliary variable is present, given the 
observed values in the sample, it introduces a Polya urn 
distribution as a pseudo posterior distribution over the 
unobserved members of the population. This pseudo 
posterior distribution can be used to obtain point and 
interval estimates of a variety of population quantities. It 
is related to the Bayesian bootstrap of Rubin (1981) and 
the Dirichlet process prior of Ferguson (1973). When 
estimating the median it yields results similar to those of 
Binder (1982). A theoretical justification for it is a step- 
wise Bayes argument which yields the admissibility of the 
resulting estimators. See for example Meeden and Ghosh 
(1983). There the admissibility of the Basu estimator was 
demonstrated. In that case the Basu estimator was shown 
to arise from a ‘posterior’ which treats the known and 
unknown ratios, r; = );/x; as exchangeable. Note that 
this is very similar in spirit to the super population model 
justification for this estimator given above, where the 
ratios r; = y;/x; were independent and identically distrib- 
uted. We shall see that the stepwise Bayes logic underlying 
the Basu estimator for the mean carries over in a straight 
forward way to point and interval estimators for the 
median. Unfortunately this is not the case for some of the 
other estimators. One natural but perhaps naive estimator 
which mimics in some sense the ratio estimator of the mean 
is just the ratio of the median of the y values in the sample 
to the median of the x values in the sample multiplied by 
the median of the x values in the population. There is no 
known model based theory which underlies this estimator 
as is the case for the ratio estimator of the mean. 

In the Bayesian approach to finite population sampling 
one needs to specify a prior distribution. Then given a 
sample, inferences are based on the posterior distribution, 
which is the predictive distribution for the unseen members 
of the population given the units in the sample. In the 
stepwise Bayes approach, given the sample one always has 
a ‘posterior’ distribution but it does not arise from a single 
prior distribution. However this ‘posterior’ distribution 
can be used in the usual Bayesian manner to find point and 
interval estimators of parameters of interest. We now will 
show how the stepwise Bayes model which yields Basu’s 
estimator for the mean can also be used when estimating 
the median. In this setup, given a sample, the predictive 
distribution for the unobserved ratios treats the observed 
and unobserved ratios as ‘exchangeable’. 

For definiteness suppose our sample contains the first 
n units of the population. We construct an urn which 
contains n balls where ball 7 is given the value of the /-th 
observed ratio, say r;. We begin by selecting a ball at 
random from the urn and the observed value is assigned 
to the unobserved unit n + 1. This ball and an additional 
ball with the same value is returned to the urn. Another 
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ball is chosen from the urn and its value is assigned to the 
unobserved unit n + 2. This ball and another with the 
same value are returned to the urn. This process is con- 
tinued until all of the unobserved units have been assigned 
a ratio. Once they have all been assigned a value we have 
observed one realization from our ‘posterior’ distribution 
for the unseen ratios given the sample of seen ratios. If in 
this process the unobserved unit / has been assigned the 
ratio with value r we then assign its y; value to be rx;. 
Hence using simple Polya sampling we have created a 
predictive distribution for the unobserved units given the 
sample. We call this predictive distribution the ‘Polya 
posterior’. It is easy to check that this predictive distribution 
gives the Basu estimator when estimating the population 
mean under squared error loss. 

Given the sample the ‘Polya posterior’ yields a predictive 
distribution for the unobserved members of the population 
and hence a predictive distribution for the median as well. 
From the decision theory point of view the usual loss 
function is absolute error when estimating a median. For 
this loss function the Bayes estimate is just the median of 
the posterior or predictive distribution for the population 
median. If one were using squared error loss for estimating 
the median then the Bayes estimate is just the mean of the 
predictive distribution for the population median. The 
admissibility of these estimators under the appropriate loss 
function follows from a stepwise Bayes argument in the 
same way as the proof of admissibility for the Basu esti- 
mator of the population mean. In Meeden and Vardeman 
(1991) and Meeden (1993) the following somewhat sur- 
prising fact was noted. For many common distributions 
the mean of the predictive distribution for the population 
median performed better than the median of the predictive 
distribution for the population median under both loss 
functions. Similar results hold for this problem. Hence our 
estimator will be the mean of the predictive distribution 
for the population median even though we will follow 
standard practice and use absolute error as our loss 
function. We will denote this estimator by estpp. This 
estimator cannot be found explicitly. However we will find 
it approximately by simulating observations from the 
posterior or predictive distribution for the population 
median. Under the Polya sampling scheme for the ratios 
described above we can simulate a possible realization of 
the entire population. For this simulated copy we can then 
find its median. If we repeat this process R times then we 
have simulated the predictive distribution of the population 
median under the ‘Polya posterior’. When R is large the 
mean of these R simulated population medians yields, 
approximately, the estimate estpp. 

In what follows we will compare the estimator estpp to 
several other estimators. Another estimator we consider 
is just the sample median of the y,’s. This ignores the 
information contained in the auxiliary variable and is used 
as a bench mark. It will be denoted by estsm. Another 
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estimator is the natural analogue of the ratio estimator of 
the population mean. This is discussed in Kuk and Mak 
(1989) and denoted by estrm. It is just the ratio of the 
median of the y values to the median of the x values in the 
sample multiplied by the median of all the x values in the 
population. They proposed two other estimators for the 
median. We will consider just the first one and denote it 
by estkm. This estimator has a plausible intuitive justifi- 
cation and can be found in their paper. Rao, Kovar and 
Mantel (1990) considered a designed based estimator for 
the median. We will denote this estimator by estrkm. 
Since this estimator can be time consuming to compute we 
will find it approximately using a method due to Mak and 
Kuk (1993). Finally we will consider the estimator pro- 
posed in Chambers and Dunstan (1986) and denote it by 
estcd. Actually Chambers and Dunstan propose a whole 
family of estimators and we will only consider one special 
case which is appropriate when u; = Xi in the super 
population model described at the beginning of this section. 
We now briefly outline the argument that leads to their 
estimator of the median. Let F denote the cumulative 
distribution function associated with the y values of the 
population. That is F puts mass 1/N on each y; in the 
entire population. The first step is to get an estimator of 
F(t) for an arbitrary real number ¢. If s denotes our sample 
of size n then given the sample we can write 
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where A(z) is the step function which is one when z = 0 
and zero elsewhere. Since the first sum in the above 
expression is known once we have observed the sample, 
to get an estimate of F(f) it suffices to find an estimate 
of the second sum. Now under our assumed super pop- 
ulation model the population ratios (y; — bx;)/Jx; are 
independent and identically distributed random variables. 
Since after the sample s is observed a natural estimate of 
bis b = Yiesyi/ ViesX; one could act as if the nm known 
ratios (y; — bx;)/Jx; for / € s are actual observations from 
this unknown distribution. Under this assumption, for a 
fixed t and a fixed unit / not in the samples an estimate of 
A(t — y,;) is just the number of the 7 known ratios incor- 
porating 5 less than or equal to (¢ — bx;) / x; divided 
by n. Finally if we sum over all the unobserved units / these 
estimates of A(¢ — y;) we then have an estimate for the 
second sum in the above expression for F(t) which then 
yields an estimate of F(t). Once we can estimate F(t) for 
any ¢ by say F(t) then the estimate of the population 
median is inf{t:F(t) = 0.5}. 


3. THE POPULATIONS 


We will compare these estimators using several different 
populations. We begin with three actual populations. The 
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first is a group of 125 American cities. The x variable is 
their 1960 populations, in millions, while their y variable is 
the corresponding 1970 populations, again in millions. The 
second is a group of 304 American counties. The x variable 
is the number of families in the counties in 1960, while the 
y variable is the total 1960 population of the county. Both 
variables are given in thousands. The third population is 
331 large corporations. The x variable is their total sales 
in 1974 and the y variable their total sales in 1975. The sales 
are given in billions of dollars. We denote these three 
populations by ppcities, ppcounties and ppsales. For the 
three populations the correlations are .947, .998 and .997. 
These populations were discussed in Royall and Cumberland 
(1981). Our ppcounties is similar to their population 
Counties60 except we have taken the x variable to be the 
number of families rather than the number of households. 

We have also considered six artificial populations. In each 
case the auxiliary variable x was chosen first and then the 
y variable was generated from it. In some cases we followed 
the super population model described at the beginning of 
the previous section for some choice of the u;’s. In some 
other cases we violated the assumption that conditional 
on the value x; the mean of y; is bx;. In all cases the errors, 
the e;’s, were independent and identically distributed 
normal random variables with mean zero and variance one. 

In the first population, ppgamma20, the x;’s were a 
random sample from a gamma distribution with shape 
parameter twenty and scale parameter one. Then given x; 
the conditional distribution of y; was normal with mean 
1.2x; and variance x;, i.e., uj; = JX. 

In the second population, ppgammaSa, the x;’s were ten 
plus a random sample from a gamma distribution with 
shape parameter five and scale parameter one. Then given 
x; the conditional distribution of y; was normal with mean 
3x; and variance x;. 

In ppgamma5b the auxiliary variable was the same as 
in ppgammaSa. Then given x; the conditional distribution 
of y; was normal with mean 3x; and variance x?. 

In ppstskew the auxiliary variable was strongly skewed 
to the right with mean 42.63, median 39.29 and variance 
204.59. Then given x; the conditional distribution of y; 
was normal with mean x; + 5 and variance 9x;. 

In ppln the auxiliary variable was a random sample 
from a log-normal population with mean and standard 
deviation (of the log) 4.9 and .586 respectively. Then given 
x; the conditional distribution of y; was normal with mean 
x; + 2 log x; and variance x?. 

In ppexp the auxiliary variable was fifty plus a random 
sample from the standard exponential distribution. Then 
given x; the conditional distribution of y; was normal with 
mean 80 — x; and variance (.6 log x;)”. 

All the populations contain 500 units except ppstskew 
which has 1,000. The correlations between the two variables 
for these last six populations are .76, .87, .41, .61, .S8 and 
— .28 respectively. 
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In most examples where ratio type estimators are used 
both the y;’s and x;’s are usually strictly positive. In 
population ppstskew 13 of the 1,000 units have a y value 
which is negative. In the original construction of popula- 
tion ppln quite a few more of the y values were negative. 
The population was modified so that all the values are 
greater than zero. 

Note that these populations were constructed under 
various scenarios for the relationship between the x and 
y variables. Ppgamma20 and ppgammasSa satisfy the 
assumptions of the super population model leading to 
estcd, while ppgamma5b is consistent with the assumptions 
underlying estpp. In ppstskew the conditional variance of 
y; given x; is consistent with estcd while for the unmodified 
ppln it was consistent with estpp. In both these cases the 
assumption for the conditional expectation is not satisfied. 
For the populations ppcounties, ppgammasSa and ppln we 
have plotted y against x and y/x against x. The results are 
seen in Figures 1 through 3. 
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Figure 1. For ppcounties the plot of y versus x and y/x versus 
x where x is the number of families (thousands) living 
in a county and y is the total population (thousands) 
of the county for 304 counties. 


The estimator estpp is based on the assumption that 
given the sample s our beliefs about the observed ratios, 
i.e., the ratios y;/x; for i€ s and the unobserved ratios, 
i.e., the ratios y;/x; for / € s are roughly exchangeable. In 
particular this means that one’s beliefs about a ratio y; /x; 
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Figure 2. For ppgammaSa the plot of y versus x and of y/x 
versus x. 
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Figure 3. For pp/n the plot of y versus x and of y/x versus x. 
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should not depend on the size of x;. In fact ppgamma5b 
was constructed so that this would indeed be true. On the 
other hand, under the super population model leading to 
the estimator estcd we would expect the variability of the 
ratios to get smaller as the size of the x variable increases 
while the average value of the ratios in any thin vertical 
strip remains roughly constant as the strip moves to the 
right. This is seen clearly in the plot of the ratios for 
population ppgammasSa. For the rest of the populations, 
except for ppgammaz20 the values of the ratios do in fact 
depend on size of x. This is seen clearly in the plots for 
ppcounties and ppln. Hence they should make interesting 
test cases for the estimator estpp. Ppexp was included as 
a test case to see what would happen if the underlying 
assumptions of estpp and estcd were strongly violated. 


4. SOME SIMULATION RESULTS 


To compare the six estimators 500 simple random 
samples of various sizes were taken from the nine popula- 
tions. For each sample the values of the six estimators were 
computed. For the estimator estpp this meant finding it 
approximately by simulating R = 500 realizations of the 
predictive distribution for the population median induced 
by the ‘Polya posterior’. In each case the average value and 
average absolute error of the estimator were computed. 
In Table | the average values of all the estimators except 
estsm are given. All the estimators are approximately 
unbiased except in one case, estcd for the population ppln. 
We did not include the results for estsm since it is well 
known that it is unbiased. In Table 2 the average absolute 
error for all six estimators are given. We see from Table 2 
that estcd and estpp are the clear winners. They both 
perform better than the other four estimators in every 
case but one. In ppexp they are both beaten by estsm, but 
this is one case where neither would be expected to do well. 
For the first seven populations their performances are 
nearly identical while for population pp/n the estimator 
estpp is preferred and for population ppstskew the opposite 
is true. 

In practice one often desires interval estimates as well 
as point estimates for parameters of interest. Kuk and Mak 
(1989) and Chambers and Dunstan (1986) each suggested 
possible methods for finding interval estimates based on 
their estimator using asymptotic theory. But in each case 
they did not actually find any interval estimators. Meeden 
and Vardeman (1991) noted how approximate 95% credible 
regions based on the ‘Polya posterior’ can be found approx- 
imately. If we let g(.025) and qg(.975) be the .025 quantile 
and the .975 quantile of the collection of 500 simulated 
population medians under the ‘Polya posterior’ then 
(q(.025), g(.975)) is an approximate 95% credible interval. 
(See Berger 1985 for the definition of such intervals.) 
Table 3 gives the average length and relative frequency of 
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coverage for these intervals. We see that for these popula- 
tions the intervals have reasonable frequentist properties. 
Perhaps this is not unexpected given the discussion in 
Meeden and Vardeman (1991). But on the other hand only 
one of the populations was constructed so that the ratios 
y;/x; are exchangeable. These results suggest that point 
and interval estimators of the median based on the ‘Polya 
posterior’ for the ratios are fairly robust against the 
exchangeability assumption and should work well in a 
variety of situations. This will be discussed further in 
section 5. 


Table 1 


The Average Value of Five Estimators of the Median 
for S00 Simple Random Samples 


Population Sample Average Value of the Estimator 


(median) Size estrm  estkm estrkm  estcd _—_estpp 
ppcities 5) .197 .196 .193 .195 195 
(1.90) 
ppsales 30 1.21 1 2S} 123) 1.25 1.24 
(1.24) 
ppcounties 30 18.21 18.60 18.66 18.26 18.39 
(18.33) 
ppexp 30 29:03, 29:05) 29:00! «29:03 «29.05 
(29.02) 
ppgammasa 30 43.82 43.88 43.91 43.99 43.89 
(43.90) 50 43.90 43.91 43.85 44.06 43.90 
ppgammaSb 30 43.84 43.96 44.19 44.15 43.61 
(44.17) 50 44.28 44.37 44.18 44.18 43.98 
ppgamma20—-:30 DBP | BARS TAB MEN ORIN RTA 
(23.15) 50 PBN ABN ~ PBs Pie easels 
ppln 30 171.15 169.38 168.12 185.01 170.61 
(170.25) 50 169.15 167.54 167.65 185.03 169.61 
ppstskew 30 43166 9 40:27) «45.88 » 45-504" 45z1 


(46.12) 50 44.04 40.70 46.01 45.43 45.37 


Table 2 


The Average Absolute Error of Six Estimators of the Median 
for 500 Simple Random Samples 


Average Absolute Error of the Estimator 


: Sample 
Population Sj 
ze estsm =estrm estkm estrkm estcd estpp 
ppcities 25 .0326 .0161 .0162 .0155 .0075 .0072 
ppsales 30 .1797 .0770 .0797 .0870 .0244 .0245 
ppcounties 30 Bal 586964 1.34 a2 1Smes 214 
ppexp 30 .43 49 48 47 48 .46 
ppgammaSsa 30 1.36 £96) P1HO3 .89 54 53 
50 95 74 .78 65 44 43 
ppgammasb 30 2 SA 4 Del DOSY WB IBS 
50 DANS PLE ILI 1.89 1.80 1.85 
ppgamma20 30 1.08 1.06 1.05 88 .67 .64 
50 .94 sf F 78 HB =) 49 
ppln 30 DSO 2 5eC 2402) 921.62) S214 ee O 
50 NSOmme2Oely weal ee MOE 4G Udi, 1287 
ppstskew 30 3.86 4.26 6.69 By PAPA BSS GS 
50 DPX BSS SLY? Mey PLL PSY 
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Table 3 


The Average Length and Relative Frequency of Coverage 
for a .95 Credible Interval for the Median Based on the 
‘Polya Posterior’ for 500 Simple Random Samples 


: Sample Average Frequency 
Populston Size Length of Coverage 

ppcities Ps. .041 .968 
ppsales 30 141 .964 
ppcounties 30 1.44 .994 
ppexp 30 2.26 .944 
ppgammasa 30 2.70 .950 
50 Dats 956 
ppgammasb 30 11.67 .932 
50 8.86 .942 
ppgamma20 30 3.24 -960 
50 72-5y\ .964 
ppin 30 84.8 934 
50 65.4 -956 
ppstskew 30 15.52 .936 
50 12.00 .938 

5. DISCUSSION 


The motivation for the estimator estpp is based on the 
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We now wish to address the question of the conditional 
behavior of the intervals for the median based on the Polya 
posterior which were developed in this note. In the simula- 
tion studies given earlier simple random sampling was used 


‘for convenience. To get some idea of the conditional 


behavior of the ‘Polya posterior’ we considered five of 
our populations. In each case we ordered the population 
using the values of the auxiliary variable x. We then took 
500 random samples from the first or smallest half of 
the population, then 500 more random samples from 
the second or largest half of the population and finally 
500 more random samples from the middle third of the 
population. We then calculated the .95 credible interval 
for the median based on the ‘Polya posterior’ which 
assumes the exchangeability of the ratios y,;/x;’s. In 
Table 4 we give the results for the ‘Polya posterior’ esti- 
mators for the median. (We also computed the average 
value and average absolute error of estcd for these 
examples. We did not include these results since they match 
closely the results of the ‘Polya posterior’.) We see that 
their conditional behavior, at least in these cases, is very 
much like their unconditional behavior. In short, interval 
estimates for the median based on the ‘Polya posterior’ 
should have reasonable frequentist properties, no matter 
how the sample was selected, as long the population approx- 
imates our beliefs that the ratios are roughly exchangeable. 


assumption that the population ratios y;/x;’s are exchange- 
able. This assumption can be described mathematically in 
two separate but related ways. The first is the super popu- 
lation model given earlier while the second comes from the 
‘Polya posterior’ which is based on a stepwise Bayes argu- 
ment and gives a non-informative Bayesian interpretation 
for the estimator. This second approach can be used no 
matter what parameter is being estimated. When esti- 
mating the mean it leads to Basu’s estimator which 
performs very much like the ratio estimator although the 
ratio estimator usually does a bit better. When estimating 
the median it leads to the estimator discussed in this note. 
Here we have argued that the ‘Polya posterior’ for the 
ratios leads to good point and interval estimators for the 
median when an auxiliary variable is present and seems to 
be reasonably robust against the assumption that the ratios 
y;/X;?S are exchangeable. 

Royall and Cumberland (1981) gave an empirical study 
of the ratio estimator and estimators of its variance. They 
argued that given a sample an estimate of variance based 
on the super population model, which leads to the ratio 
estimator, often made more sense than a design based 
estimate based on a probability sampling distribution. In 
Royall and Cumberland (1985), they demonstrated that, 
conditional on the sample mean of the auxiliary variable, 
the conditional coverage properties of the usual designed 
based confidence interval for the population mean were 
‘hopelessly unreliable’. 


Table 4 


The Average Value and Absolute Error for the Point 
Estimator and the Average Length and Relative 
Frequency of Coverage for a .95 Credible Interval 
for the Median Based on the ‘Polya Posterior’ 
for 500 Simple Random Samples from the whole 
Population, the ‘Smallest’ Half, the ‘Largest’ 
Half and the ‘Middle’ Third 


: Sample Where Average Average Average Frequency 
Pomuauon Size Taken Value Error Length of Coverage 
ppcities 25 whole 195 .0072 041 .968 
smallest .192 .0047 033 -994 
largest 2 .196 .0078 .048 .988 
middle % .201 .0114 055 -922 
ppcounties 30 whole 19.4 .220 1.46 -990 
smallest 2 18.6 -305 1.34 -942 
largest 2 18.1 283 1.59 -954 
middle 4% 18.5 psy? 1.35 -964 
ppsales 30 whole 1.24 0072 141 -964 
smallest “2 1.24 027 oS -966 
largest 2 1.23 .020 M25 -982 
middle 4% 1.23 .027 5139 944 
ppgammasSa 30 whole 43.9 53 2.70 -950 
smallest “2 43.8 35 2.82 948 
largest 2 44.0 53 2.55 940 
middle % 43.9 47 2.63 974 
ppgammasb 30 whole 43.6 2.38 117 .932 
smallest 2 42.2 2.69 11.6 .890 
largest “% 45.1 D5 a .2 -950 
middle 45.2 2.27 113 .936 
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As can be seen by looking at the plots of y;/x; versus 
x; and our simulation results it does not seem to matter 
much if the variability in the ratios y;/x;’s decreases as x; 
increases. What is crucial however is that the average value 
of the ratios in the narrow strip above a small interval of 
possible x values remains fairly constant as we move the 
small interval to the right. In Figure 2, the plot of the ratios 
for ppgammaSa is an example of such a plot. In fact this 
is how the population was constructed, since it satisfies the 
assumptions underlying estcd. In Figures 1 and 3 we see 
for ppcounties and ppln that the average value of the ratios 
in a narrow strip tends to decrease as we move to the right 
and helps to explain the relatively poorer performance of 
the ‘Polya posterior’ estimators in these cases. Overall 
however, the performance of procedures based on the 
‘Polya posterior’ seem to be reasonably robust against the 
exchangeability assumption. 

As another alternative we could consider a more balanced 
sampling plan which is based on stratifying the population 
on the auxiliary variable. For example consider again 
population ppgamma5b where it is ordered on the basis 
of its x; values. We constructed ten strata where the first 
stratum consisted of the units with the fifty smallest x; 
values, the second stratum of the units with the next fifty 
smallest x; values and so on. We then took 500 stratified 
random samples of size fifty where five units were chosen 
at random from each stratum. For these samples the 
average value of estpp was 43.94 and its average absolute 
error was 1.81. The average length of its corresponding 
interval estimator was 8.95 with .938 relative frequency 
of covering the true value. Note that these figures are very 
similar to those given Tables 1 and 2 when simple random 
sampling was used. 


ACKNOWLEDGEMENTS 


Research supported in part by NSF grant SES 9201718. 


REFERENCES 


BASU, D. (1971). Anessay on the logical foundations of survey 
sampling, part one. In Foundations of Statistical Inference. 
Toronto: Holt, Reinhart and Winston, 203-242. 


(ai, 


BERGER, J.O. (1985). Statistical Decision and Bayesian 
Analysis. New York: Springer-Verlag. 


BINDER, D.A. (1982). Non-parametric Bayesian models for 
samples from finite populations. Journal of the Royal 
Statistical Society, Series B, 44, 388-393. 


CHAMBERS, R.L., and DUNSTAN, R. (1986). Estimating 
distribution functions from survey data. Biometrika, 73, 
597-604. 


FERGUSON, T.S. (1973). A Bayesian analysis of some non- 
parametric problems. Annals of Statistics, 1, 209-230. 


KUK, A.Y.C., and MAK, T.K. (1989). Median estimation in the 
presence of auxiliary information. Journal of the Royal 
Statistical Society, Series B, 51, 261-269. 


MAK, T.K., and KUK, A. (1993). A new method for estimating 
finite-population quantiles using auxiliary information. The 
Canadian Journal of Statistics, 21, 29-38. 


MEEDEN, G., and GHOSH, M. (1983). Choosing between 
experiments: applications to finite population sampling. 
Annals of Statistics, 11, 296-305. 


MEEDEN, G., and VARDEMAN, S. (1991). A noninformative 
Bayesian approach to interval estimation in finite population 
sampling. Journal of the American Statistical Association, 
86, 972-980. 


MEEDEN, G. (1993). Noninformative nonparametric Bayesian 
estimation of quantiles. Statistics and Probability Letters, 16, 
103-109. 


RAO, J.N.K., KOVAR, J.G., and MANTEL, H.J. (1990). On 
estimating distribution functions and quantiles from survey 
data using auxiliary information. Biometrika, 77, 365-375. 


ROYALL, R.M. (1970). On finite population sampling theory 
under certain linear regression models. Biometrika, 57, 
377-387. 


ROYALL, R.M., and CUMBERLAND, W.D. (1981). An 
empirical study of the ratio estimator and estimators of its 
variance. Journal of the American Statistical Association, 
76, 66-88. 


ROYALL, R.M., and CUMBERLAND, W.D. (1985). Condi- 
tional coverage properties of finite population confidence 
intervals. Journal of the American Statistical Association, 
80, 355-359. 


RUBIN, D.B. (1981). The Bayesian bootstrap. Annals of 
Statistics, 9, 130-134. 


air 


AdimetOH ane sora ete reel bs cae ; 
Osa id ‘ - eit , 
Siw ee ia ry a a te rs act eee 


hiad iv Ai MY Lb ace nt terrtie 

octets, SAA Ash {oc eit if wT, Serie apedio'ss 
— apa oa ee eee othe Aiea ve 
 eetintt aehy ee es SEWALT arity aaa By deiutecadtndl 
Co piituno® amb orn fod aoa echeiigs Spear tah Gye eae, “tate 
; 1. wont ag Ne Te 


ig \ . : : ¢ . 
ary Oy: to devinns aslaoyed A. as “Ome ~ tiie rere earl nt’ ay : 
ee Jf EAMES, Yo Monin ig leva. t py f “ae con Aas zn hl? aa re 


it rt Enelsctiines ribgiA (ety BT sthapian, 2% aiktta 
\rdA ott \o hawweak, momeunriial wrai lta to lg 

wag-fat 12 8 erie Aysicvee Woon 
iw | fag > 


qotnnting fo) bodsedtwer A CERRY A “903 bein . ee HAM 
wA Le AY IP ninbal eoadiege eset pest zie, pinta Sei? 
Reset 1S ,coesehvot?. ns eel, an ore 


paeeist golimewhl tte tb Neon ‘hes rot wadism 
giiqnar VotmagGE a nr) i! eooljiendge comane rots 
Cy a, tL ine o ene 
‘| a ; 


sv}hurrotn tert“ Omer 2 MAPA F bee CF, AR 
goltaluney wlaft AL anueniteleerianind tines nok sakeniee in, 
ony taleoe A Versi rold, remefors dh GAN ie Yonah -gatitngrees 


ioe Lee ia 


ante aripie equan wiremp halo (etl) a7 MadaaM 
Al qe Dede aa etieinr aries Uy aaa be . 


tl ai) ae od) eigen set! 104 Jinuiere oer at 
i Bie siulowte aguraverati bare be, thine nent 
nts gINIS) 1h pe Aa -% tA vee ye a zh gata: ws a6 8 or 


ryt ney fest hed nes anni add Love hse) qouseeices ¥ f sent Z r 
APEC oO nda parler tint orks Bee, oe tae ’ ‘ 
oe Paget liey flea icar | Bi pore INTE a PER Aa IO se at pee tz et : 
oF yi Gr Poll ees de SR -ae SY 4) nae § 58 Fieve US hol vrs 7 by bie get 
' retail gt ae hy is GE b pana 6 Pepe@een liq “eaylo* Halt, Sap Ae - We 
ri (aie Sut ond tre “dais Ths < A 
aA TRU ow Avail bas” i x tar a in ery, ate _ 
doles -\ Gar wikis iter SN Ve Gali iwi tigi — si : na , a ‘" 
vest yer A Tee lhe ial aay Ye ENE te ihe e va oa Sveti B 


rT waa” at BITTON ZIT GR aM vo a ToT riejua fio 


i 
gn. ty a ee Pree ers ra ’ plod 7 af 
pA e Ear ete Pit hare e Wor ii AYO one =r aanih 

sede. (iS Myke imreay ier) $2) Al meg myer ta tee wert . ARE ey a 
WONG GEIS Wate ie Wits Wl NG rai wean - 
| rv . P » ’ “uy Ae Pad a hy ' wl “ ota tees “ 
. ty open rund eviienalhaiteiodhan wenmh Re. 
\p Ani, megiegay eee ortT enh 22 waa CTA! Se iiadier Wie 5) Sha 5 
KET OC) © edyebait acne hehe as as 


‘ : J ‘ 6° Sire Gueer a a GA 
9 7@ ©mo 
15 y nail « tf ¢ aATIaT SS Age y ov : 
Lenina 12> )ased ah 
1 alg, Wheel (ties 19-0 PA oe 
( e fish 2 OCitee 2.494) ° = >" ~ =e 
noolte’ Goninon, en 2 
i : wien ' 
uf 4ne¢ (saben wets, ¢ ethowmuceled Lut ; eee 
i i” > F Files ve tae 
; i ; : : - ary var Wal om) : . @ ivglh 
; “hiee< sO Wl bikg A ee om if}.< é 
; ti: memutaude Socods Mic are 
= 4) et. & 
: x ev t sige 
4 


Survey Methodology, June 1995 
Vol. 21, No. 1, pp. 79-87 
Statistics Canada 


19 


Outlier Robust Horvitz-Thompson Estimators 


BEAT HULLIGER! 


ABSTRACT 


The Horvitz-Thompson estimator (HT-estimator) is not robust against outliers. Outliers in the population may 
increase its variance though it remains unbiased. The HT-estimator is expressed as a least squares functional to 
robustify it through M-estimators. An approximate variance of the robustified HT-estimator is derived using a kind 
of influence function for sampling and an estimator of this variance is developed. An adaptive method to choose 
an M-estimator leads to minimum estimated risk estimators. These estimators and robustified HT-estimators are 
often more efficient than the HT-estimator when outliers occur. 


KEY WORDS: Outlier; M-estimator; Adaption; Population mean; Sampling; Sensitivity curve. 


1. INTRODUCTION 


The mean of a variable over a finite population is an 
important indicator. Examples are the mean salary of 
employees in a branch of the economy or the mean yield 
of corn of the farms in a region. Due to its connection to 
the sum the mean cannot be easily replaced by other indi- 
cators. But the population mean is a sensitive characteristic 
because a single large observation may determine its value. 
The Horvitz-Thompson estimator (HT-estimator) is a natu- 
ral estimator of the population mean if the sample design 
has unequal inclusion probabilities and is without replace- 
ment. It is the sample mean in simple random sampling. 
It is always unbiased whatever the population distribution 
of the investigated variable is. But the HT-estimator is not 
robust against outliers because it is linear in the observed 
values like its estimand, the population mean. Large obser- 
vations together with small inclusion probabilities have a 
particularly large influence on the HT-estimator. 

Suppose there is an outlier in a sample. The outlier may 
be a correct observation from the target population. 
Discarding such a correct outlier makes the HT-estimator 
biased. But keeping it with full weight makes the HT-esti- 
mator highly variable because typically the outlier would 
show up only in a few of the possible samples. Thus there 
is a tradeoff between bias and variance in this case, which 
in particular includes asymmetric distributions with one 
heavy tail. 

The outlier may also be an incorrect observation, e.g., 
due to a measurement or coding error or stemming from an 
element outside the target population. In that case keeping 
the outlier with full weight may entail a large bias of the 
HT-estimator in addition to high variability. Thus discard- 
ing incorrect outliers reduces both bias and variance. 

Since it is often difficult to detect outliers and to decide 
whether it is correct or not one would like to have esti- 
mators that perform well in terms of bias and variance 


irrespective of the nature and the detection of possible 
outliers. HT-estimators which are robustified through 
M-estimators are promising candidates for this difficult 
task. 

In the survey sampling literature the problem of outliers 
or aberrant values is often treated under the heading 
“‘skew populations’’. Kish (1965, sec. 11.4 B) describes the 
problem in economical surveys and surveys of individuals. 
He proposes the formation of separate strata for outliers 
if possible, truncation, transformation or modelling. The 
idea of forming a separate class for large units and com- 
bining the class means is investigated for example in 
(Glasser 1962) and (Hidiroglou and Srinath 1981). 

The truncation idea is made more precise by the win- 
sorized mean proposed by Searls (1966). Fuller (1991) 
proposed a preliminary-test-estimator which reduces the 
impact of the largest data values only when a test for 
extreme values is significant. Rivest (1993) studied the 
behavior of various winsorization schemes under simple 
random sampling. Shoemaker and Rosenberger (1983) 
derive exact formulae for the expected value and variance 
of the median and trimmed mean under simple random 
sampling without replacement. Oehlert (1985) proposes 
the random average mode estimator to estimate the mean 
of finite populations in an outlier robust way. Smith (1987) 
emphasises that it is as important to detect and treat 
influential observations if the inference is based on the 
randomisation provided by the sample design as if the 
observations are considered realisations of random 
variables. He proposes an influence measure for linear 
estimators based on case deletion, which involves both the 
variable of interest and its weight. 

The prediction approach in sampling theory uses 
stochastic models for the population to predict the total 
of the present realisation. Linear models and (nonrobust) 
linear estimators are used. Aspects of the sensitivity and 
robustification against model misspecification are reviewed 
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in (lachan 1984). Chambers (1986) develops an outlier- 
robustification of the prediction approach using M-esti- 
mators. He distinguishes representative and nonrepresen- 
tative outliers in a sample. Representative outliers must 
be included with full weight in an unbiased estimate of the 
population mean while nonrepresentative outliers should 
be downweighted or discarded. 

Little and Smith (1987) treat outliers and missing data 
in certain positive multivariate continuous data by a robus- 
tified EM-algorithm. Gwet and Rivest (1992) investigate 
resistant ratio estimators under simple random sampling 
without replacement. 

M-estimators form a class of flexible and simple robust 
estimators. An M-estimator T of location is defined implic- 
itely by the estimating equation 


yy W(X) — T) = 0 


i=1 


for a predetermined function y, e.g., WHu(xk) = 
max(—k,min(k,x)), where k is a tuning constant. An 
M-estimator may be written as a functional of the empir- 
ical distribution function. The influence function of an 
estimator is a functional derivative of the estimator 
(Hampel 1974). It describes the reaction of the estimator 
to a small contamination in the data. An M-estimator with 
bounded y-function usually has a bounded influence 
function such that outliers cannot disturb the estimator 
too much. For the estimation of the mean of asymmetric 
finite populations M-estimators must be adapted. 

In this article we develop design-based M-estimators for 
samples with unequal inclusion probabilities. The simple 
linear model which implicitely is the basis of the Horvitz- 
Thompson strategy is made explicit and the HT-estimator 
is expressed as a functional of an empirical distribution 
function which accounts for the complex sample design. 
This establishes the link to classical robust statistics and 
allows a straightforward robustification of the HT-estimator 
(Section 2). We define an influence function for sampling 
which clarifies the outlier-sensitivity of the HT-estimator 
and leads to an approximation of the sampling variance 
of the robustified HT-estimator. An estimator of this 
variance is presented. In Section (3) we briefly comment on 
stratification, domains, robust designs and one-step esti- 
mators. In Section (4) an adaptive robustification of the 
HT-estimator is developed. The method chooses from a class 
of robustified HT-estimators the one which minimizes an 
estimate of the mean squared error. The resulting estimator 
is called minimum estimated risk estimator (MER-estimator). 
A Monte-Carlo simulation is presented in Section 5. 
Robustified HT-estimators and MER-estimators outper- 
form the HT-estimator in many outlier situations. The 
premium to pay is a moderate loss of efficiency in situa- 
tions where the HT-estimator is optimal. 


Hulliger: Outlier Robust Horvitz-Thompson Estimators 


2. ROBUSTIFICATION OF HORVITZ-THOMPSON 
ESTIMATORS 


2.1 The HT-Estimator as a Least Squares Functional 


A finite population U = {1,...,N}of0 < N< @ 
distinct elements is sampled. We are interested in a variable 
y which takes the values y; for i € U. The sample design 
p(S) on the space of samples S of fixed size n has inclusion 
probabilities 7; = P[ié S] = Ys3;p(S). These z; are 
proportional to some known positive auxiliary variable 
x;(i € U). Such sample designs are called IPPS designs 
(inclusion probability proportional to size) because often 
x; is some size measure. Denote by 7; the joint inclusion 
probability P[i € S, 7 € S] (i,j € U). The vector of all 
y-values is denoted yy: = (1, ...,¥n) "and xyis defined 
in an analogous way. The vector of the y-values of a 
sample S is denoted ys: = (Yi,, ---+¥i,) (ix € S). The 
goal is to estimate the population mean of the variable 
Y: Ju: = Vievi/N. 

The HT-estimator for Jy is Tyr: = Viesy;/ (Na). The 
variance of 7}, is estimated by the well known estimator 


Var(Tur) = 


1 yi Yi Di 
al (1 HM aad Syl Getic LATS Tea 


ieS P ixjeS bal cae 
(1) 


which is due to Horvitz and Thompson or by the variance 
estimator due to Yates, Grundy and Sen (see Cochran 
1977, p. 261). 

The rationale behind the HT-estimator given in the 
survey sampling literature is that it has sampling variance 
zero if the inclusion probabilities 7; are exactly propor- 
tional to y;. Then 7y7(¥s) = Jy for every sample S. The 
HT-estimator is bias-robust but not variance-robust with 
respect to deviations from proportionality between y; and 
m7; (cf. Rao 1966). 

How can the HT-estimator be formulated in a way 
which allows the derivation of an influence function 
analogue and a variance estimator? The key idea is to 
express the HT-estimator as a least squares (LS) functional 
of an estimate of the population distribution function in 
such a way that the design is incorporated in the estimator 
of the population distribution function while the propor- 
tionality of y; and x; is taken up by the LS-functional. 

The joint population distribution function of two 
variables (x;, y;) is defined as Fy (r,t) = Viey1l{x; <= 7} 
1{y; < t}/N, where 1{y; < t} = lif y; < tand Oelse- 
where. There are various possibilities to estimate Fy but 
the easiest and most generally applicable estimator is the 
sample distribution function 
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Rs(r,6)scs0)5 14x, <rjl{yi< | y - (2) 


reson ies 
The estimator F's is a distribution function itself. 


To derive a LS-functional the following superpopulation 
model for the proportionality between y; and x; is used: 
We assume that y,yis a vector of realisations of independent 


random variables Y, with expectation 8x; and variance o7x;. 


Definition 1. The LS-estimator 6,5(Fs) of 6 in the above 
model with respect to the sampling distribution function 
Fs of (x;,y;) (i € S) minimizes |(y — Bx)*/x dF (x,y) or 
equivalently solves 


ge XN 
ye — e=) = 0. (3) 
ies. Mi 1x; x; 


The following statement is well known and its proof is 
easy. If Sis asample drawn according to an IPPS sample 
design with inclusion probabilities 7; = nx;/YVicyx; (i€ U) 
then the HT-estimator is Ty7 = XyB,s5(F's), where 
By s(Fs), the LS-estimator defined by (3):, is given by 


BR (Fe) Lies Vi/ 
Lies Xi/T; 

Note that the expression Ty7 = XyBzs5(F's) = Xy 
(Dies ¥i/7i)/(XiesXi/7;) does not depend on the super- 
population model. However the superpopulation model 
clarifies the role of the HT-estimator: The slope 6; 5(Fs) 
involved in the HT-estimator is a weighted least squares 
estimator that incorporates the information in the design 
through Fs; as well as the information in the auxiliary 
variable through the regression. 


2.2 The Robustified HT-Estimator 


After the separation of design and auxiliary information 
and its expression as a LS-functional the robustification 
of the HT-estimator is analogous to the robustification of 
LS-estimators in linear models for infinite populations 
through M-estimators (cf. Hampel et a/. 1986, Chapter 6): 
The estimating equation (3) now involves some function 7 
which depends on the standardized residuals (y; — 6x;)/ 
x}/* and on x;. For ease of notation denote by a prime the 
division by x!/? and let r’(B) = (y —Bx)/x!”. 


Definition 2. Let 8(Fs,7) be a solution of the equation 
1 tA , / 
ee er 7 (8) ) 4, — 0. (4) 
: Tj 
ieS 
The robustified HT-estimator (RHT-estimator) is 


Trur(Fs) : = XuB(Fs,n). 
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B(F’s,7) is called the slope of the RHT-estimator. 


In general useful choices of 7 are of the form n(x,r) = 
w(x)W(r + u(x)), where w(x) and u(x) are two weighting 
functions and y is a defining function for a location 
M-estimator (cf. Hampel ef a/. 1986, p. 315). In the following 
we use the so-called Mallows form, which sets u(x) = 1. 
Mallows-type regression downweights outlying x-values 
and outlying residuals independently. A well-known 
example, which also sets w(x) = 1, is the Huber-function 
n(x,r) = vouw(.k4) = max(—k, min(k,r)) for some 
constant k. The RHT-estimator with defining function 
n(x,r) = r Vv xis the HT-estimator. Thus by adjusting the 
tuning constant k in the Huber-function a smooth transition 
of estimators from the HT-estimator to more and more 
robust estimators is possible. 

Scale estimates are needed in w(x) and ¥(r) to make 
B(Fs,n) scale equivariant. While for the weighting function 
w(x/) preliminary scale estimators are available, e.g., 
the median of the x;, the scale of the residuals must be 
estimated simultaneously with the slope 8. The median of 
the absolute residuals may be used. In the following theo- 
retical development (Sections 2.3 to 4) scale is assumed 
known to simplify the treatment. 

The RHT-estimator is a nonparametric estimator. The 
model Ey = Gxis merely used to motivate the expression 
of the HT-estimator as a least squares functional. Neither 
the HT-estimator nor the RHT-estimator need this model 
or symmetry of errors with variance proportional to x in 
order to be applied. 

Other formulations of the HT-estimator as least squares 
functionals may be appropriate in certain conditions. 
Suppose that in spite of the IPPS-design y; is not correlated 
with z;. Then one would probably choose the unweighted 
sample mean Jy = Vices y;/n as an estimator of the popu- 
lation mean (cf. Rao 1966). A robustification of ~; could 
be a solution f of ViesW(y¥; — w) = 0. This is a location 
M-estimator. If the HT-estimator is in fact appropriate 
due to the correlation between y; and z; then this robus- 
tification is not efficient. 

A third robustification would assume y; proportional 
to x; but with variance proportional to the square of x;. 
This is in fact the situation where the HT-estimator is 
optimal. The corresponding robustification would be a 
solution B of Vies n(X;,¥;/X; — B) = 0. Obviously this 
robustification does not account for the IPPS-sample 
design. If the design is put back into the estimating equation 
by solving Yiesn(X;,¥i)/X; — B)/m; = O then we do not 
get back the HT-estimator when y(x,r) = r. 

One may argue that in fact the HT-estimator is never 
used in its pure form for estimating population means. The 
usual estimator is (VY jes¥;/7;)/( Vics 1/7;), sometimes 
called the Hajek-estimator. The estimating equation of the 
Hajek-estimator, Vjes(¥; — T)/a; = 0, makes obvious 
that the Hajek estimator is not robust against outliers in y. 
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But the residual y; — T does not involve the auxiliary 
variable x;. Therefore the Hajek-estimator does not suffer 
from a possible combined effect of large y; together with 
small x;, which may be a leverage point for the regression 
model underlying the HT-estimator. 


2.3 A Sampling Sensitivity Curve 


The derivation of an approximate sampling variance of 
the RHT-estimator (see Section 2.4) uses a finite population 
analogue to the influence function for infinite populations 
(Hampel 1974). For finite population sampling with design 
based inference it is appropriate to develop a sensitivity 
curve (SC) (cf. Hampel et al. 1986, p. 93) for B(F,n) at the 
population distribution function Fy. In other words, the 
slope of the RHT-functional is linearized around Fy. 
Denote by U+ the population U augmented by a unit with 
characteristic (x,y). Denote by \(8,Fy) the function 
Vieun(x/ ,r/ (B))x/ /N, such that the defining equation 
for B(Fy,n), the M-estimator at the population distribu- 
tion function, is \(8,Fy) = 0. Clearly 


(N+ DING Goa: a) hos). = AC Hy.) Fa) l=: 


Using a linear approximation to n(x, - ) and neglecting 
terms in 1/N the sensitivity curve of B(Fy ,n) can be iso- 
lated from this equation: 


(N + 1)(6(Fu4,0) — B(Fusn)) = 


A ORe ey hs 
, / 72 
Vieu 12 (Xj 517 )xj°/N 


=: SC(x,y, Fy.) (5) 


where 72(x,r) = 0n(x,r)/dr and both r’ and r/ are eval- 
uated at 6(Fy,n). This SC may be extended to the case 
of a p-dimensional explanatory variable (cf. Hampel et al. 
1986, p. 316 and Hulliger 1991, p. 183). 

Since units usually are not independently included into 
an IPPS sample, the reaction of the RHT-slope to a partic- 
ular observation must be investigated by conditioning on a 
particular sample. The deviation of the estimator 6 (Fs, 7) 
at a particular sample S from 6(Fy,7) may be approx- 
imated by integrating the SC of 8(F,7) with respect to 
the sampling distribution function Fs; (cf. Hampel et al. 
1986, p. 85): 


B(Fs,n) — B(Fu.n) = | SC(x,y,Fu,n)dFs. (6) 
The influence of unit iin sample S may then be defined 


as the contribution of the unit / to the deviation due to the 
sample S, /.e., 


SC((x%;,7;,¥;)) | S,Fu.n) = 


MEX Ge fs 
(Lyes 1/4j) Djeu 2 (Xf 0) )X/7/N 


(7) 
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The SC may be studied theoretically to discuss the 
properties of the RHT-estimator and to choose a good 
n-function. And it may be estimated by replacing the 
standardization factor N/(¥ jeune (Xj sj ) Xj *) by an appro- 


‘priate estimator. The estimated SC may be used as a tool 


for outlier detection. 
The influence of unit /in sample S on the HT-estimator is 


Xy SC ( (x;,7;,y;) |S,Fy.n = r) — 


(Vie astris (1 + 4; Nb, Inj). 


JESNI 


This SC is unbounded in y; such that the HT-estimator 
is not robust against outlying y;. The y; influences the 
HT-estimator through the residual y; — 6, 5(Fy)x;. This 
makes clear why a large y; combined with a small x; (or 
small z;) has a large influence. If 7; is directly propor- 
tional to x;, as the IPPS design in principle requires, then 
the SC of the HT-estimator is bounded in x;. In other 
words the HT-estimator is robust against outlying x;. 
However the bound may be quantitatively too high to be 
efficient and further downweighting of outlying x; may be 
necessary. 


2.4 Approximate Expectation and Variance 


Along the lines of the proof of proposition 2.1 in 
Gwet and Rivest (1992) it can be shown that 6 (F's, 7) is 
consistent for B(Fy,n) in the sense that for a growing 
and nested sequence of populations and IPPS samples 
limy. 7-oP | | B(Fs,n) i? B(Fu,n)| < €] = Ilve>0. 

Due to the consistency of 6 (F’s;,7) the sampling expec- 
tation E; 6 (F's5,n) is approximately B(Fy ,n). Of course 
Xy B(Fy.n) may be different from the population mean 
and then x, 6 (Fs,7) has a bias as an estimator of jy. In 
particular if the population distribution is not symmetric 
then Xy 6 (F's,7) is in general a biased estimator for Jy but 
nevertheless consistent for Xy8(Fy,n). The important 
question then is how large is the bias of *yG(Fs,7), in 
particular when compared with the variance. 

The SC (5) may be used to derive a variance approx- 
imation. The derivation is analogous to the case of inde- 
pendent identically distributed random variables with the 
influence function replaced by the sampling SC. Taking 
the expectation of the square of (6) one gets after some 
approximations 


Vars B (Fs,7) 


U 


Esl (8(Fs,n) — B(Fyu,n))7] 


Vate() sn(¥; 50; )% (ay) 
(Lieu 92(x/ .7/ )x4*)? 
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Tij / y / , p- tp 
ala 1) n(x Vi Xi NX; 17 )X; 


iW; 


, (8) 


pe 2 2 
Vic n2 Xioki ) Xi Ah Disjeu 12 (Xi 77 Xi ~ 2 (Xj 77) Xj 


where 7/ is evaluated at 8 (Fy ,n). Denote this approximate 
variance by V,. An important difference to the case of the 
asymptotic variance of an M-estimator with independent 
identically distributed random variables is that the cross- 
product terms in the numerator of V, do not vanish. If 
n(x,r) = r then V, yields the correct variance of the 
HT-estimator. 


2.5 Estimation of the Variance 


The numerator of V, is the variance of Yiesn(x/ ,r/ ) 
(B (Fy ,n))x/ /x; which is a HT-estimator apart from the 
unknown 7; (6 (Fy ,n)). Therefore the variance estimator 
(1) for the HT-estimator may be used. After replacing 
B(Fy ,n) by the estimator 6 (F's5,7), the estimator of the 
variance of the RHT-estimator becomes 


1 
Lies a 
2 ny 
Uipder > sacha, 


Therefore different robustifications may be appropriate 
for estimating stratum means and overall means. 

This is a general problem for robust estimation in sub- 
populations (domains) since the definition of an outlier 
depends on the reference population. An observation may be 
an outlier in a particular subpopulation but may be harmless 
in another one. Thus a robust estimator may be suited for 
one subpopulation but perform poorly in another subpopula- 
tion. Often no robustification is needed or wanted for overall 
means but subpopulation means need to be robustified 
because of outliers that turn up. Luckily the sample size is 
often considerably smaller in a subpopulation than in the 
whole population and then the bias component of the MSE 
of a robust estimator is often smaller than the variance 
component. Thus robust estimators may be more efficient 
than the HT-estimator when used in domain estimation. 


1 
/, 2: / / 
(Xj 17) XE A Viszejes — 00%] 07 Xi (Xj 77 )X; 


if 


(9) 
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The minus sign in (9) is in order. The (negative) cross- 
product terms in the numerator usually dominate. Never- 
theless v,+;7 may become negative as can the HT-variance 
estimator (1) itself (cf, Cochran 1977, p. 261). The variance 
estimator v,;,7 does not yield the variance estimator (1) if 
n(x,r) = r. Of course the Yates-Grundy-Sen estimator may 
be used to estimate the numerator of V,. A third variance 
estimator may be derived by writing the RHT-estimator 
as a weighted least squares estimator whose weights depend 
on the estimate (cf. Hulliger 1991, p. 166). Since the MER- 
estimators (cf. Section 4) performed slightly better with 
v,7 than with the other variance estimators the simulations 
of Section 5 were done with v,#7. 


3. EXTENSIONS 


3.1 Stratification and Domains 


The stratified mean under stratified random sampling 
is a HT-estimator. The stratified mean may be written as 
the mean of predicted values under a one-way analysis of 
variance model. The corresponding robustification is 
straightforward. It amounts to the separate robustification 
of the stratum means (Hulliger 1991). However, if the 
stratum sample size is 1 or 2 no outlier can be down- 
weighted without the help of further assumptions. Further- 
more the biases of the robustified stratum means may add 
up to a large overall bias (cf. Rivest 1993, Section 4). 


ij 


3.2 Hansen-Hurwitz Strategy 


When sampling is done with replacement and with 
unequal drawing probabilities the Hansen-Hurwitz estimator 
is used instead of the HT-estimator. The Hansen-Hurwitz 
estimator may be robustified analogously to the HT- 
estimator (see Hulliger 1991, section 4.4) since the underlying 
model is the same. The variance approximation for the 
robustified HH-estimator is simpler than for the RHT- 
estimator because the crossproduct terms vanish due to the 
drawing with replacement of the Hansen-Hurwitz design. 


3.3 Robustified IPPS Design 


The ratios y;/a; in the HT-estimator act like the sum- 
mands of an arithmetic mean. Small z; together with large 
y; inflate the HT-estimator. To robustify the design against 
very large and very small inclusion probabilities we may 
put 7; = nx;/ Dux where X; = Xy Ste Wau (X; ca Xy,k). 
Thus the auxiliary variable x; is ‘‘Huberised’’ from its 
mean to prevent too high and too low values. Now an 
IPPS sample is drawn with inclusion probabilities 7;. The 
HT-estimator is still Ty7 = (1/N) ¥sy;/7; and it is still 
unbiased. Of course it is not robust against outliers in y 
and it may loose efficiency if the expectation of the y; is 
not proportional to 7;. The weighted LS-estimator under 
the superpopulation model for the HT-estimator (see 
Section 2.1) with inclusion probabilities 7; and unmodified 
auxiliary variable x; is 
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Ys ¥i/ Tj 


lik) = ; 
Brs(Fs) ¥ 6X/¥, 


(10) 


with corresponding estimator for the population mean 
Xy Bis (Fs). This 8,5 may be robustified against outliers 
in y; like the HT-estimator. Ratio estimators in IPPS 
samples are of the same form with the original z; instead 
of z;. Thus ratio estimators may be robustified anal- 
ogously to HT-estimators, too (cf. Gwet and Rivest 1992). 


3.4 One-step Estimators 


It is not advisable to express robust estimators as weighted 
means with fixed weights attached to the observations 
because the notion and the effect of an outlier depend on 
the particular domain and variable to be analysed. How- 
ever, so-called one-step estimators, which are expressed 
as weighted means, reduce the computational complexity 
of robust estimators. The one-step RHT-estimator is 


ALLA ip (11) 

LiesWiX) “/ 7} 
with weights w; = 9(x/,¥/ — Brsxi)/(¥i — Brsxj). In 
fact this is the result of the first step of the iteratively 
reweighted least squares algorithm, which is often used to 
calculate M-estimators. The one-step RHT-estimator 
inherits much of the good properties of the fully iterated 
RHT-estimator and is simpler to implement and faster to 
compute. 


4. MINIMUM ESTIMATED RISK ESTIMATORS 


The RHT-estimator is in general biased. A convenient 
performance criterium is the sampling mean squared error 
(MSE) Es[ (%y B(Fs,7) — ¥y)?]. For small to moderate 
samples the gains of RHT-estimators over the HT-estimator 
are not very sensitive to the particular robustification 
chosen if there are outliers in the sample (cf. Hulliger 1991, 
Chapter 3). But with well-behaved data or for moderate 
to large samples the losses in MSE of certain RHT-esti- 
mators may be considerable. The question arises how to 
choose a good RHT-estimator. Minimum estimated risk 
estimators (MER-estimators), which adapt the tuning 
constant of a RHT-estimator to the sample, are a possibil- 
ity. MER-estimators for the expectation of a univariate 
random variable are investigated in Hulliger (1991, 
Chapter 2). The idea is to take a simple M-estimator like 
a Huber M-estimator, to estimate its MSE for a set of 
tuning constants k, and to choose the tuning constant with 
least estimated MSE. 

Huber’s (1964, p. 97) proposal 3 and Jaeckels (1971) 
adaptive trimmed mean aim at symmetric random variables 
and therefore use a variance estimate instead of an estimate 
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of the MSE. MER-estimators are similar but their aim is 
to estimate the mean of asymmetric distributions. 

Here we introduce MER-estimators for IPPS designs. 
Consider a parametric set of functions {,(x,r): k € K}, 
where K C R% is the set of parameters. Usually p = 1 
or 2 to make minimization feasible and to keep the effi- 
ciency loss due to the estimation of the nuisance parameter 
k low. We do not call k a parameter but a tuning constant 
to avoid any confusion with the concept of parameters in 
probability distributions. A suitable set of »-functions 
induces a setB:= {B(Fs,n,) :k € K}, where B(F's, nx) 
is the slope of an RHT-estimator. To ensure consistency 
of the MER-estimator let lim,_.n,(x,r) = rv(Xx,r) 
such that the HT-estimator is an element of 8. The MSE 
of B(Fs5,n,) may be estimated by 


r(Fs,k) = max (v,(F's,k),0) ae (B(Fs,k) 7 Bis(Fs))’, 
(12) 


where v,(F’s5,k) is the variance estimator (9) or some other 
estimator of the variance of 6 (F’s,;,). We use max (v,,0) 
in r(F;,k) because the variance estimator (9) may become 
negative. Typically the function r(F;,k) with k € R, has 
a maximum at or close to k = 0 which stems from a large 
bias. Then it drops to a minimum where bias and variance 
are both small. For large tuning constants r(Fs,k) 
approaches the variance of the HT-estimator, usually 
from below. 

Definition 3. Suppose r(Fs,.) has a global minimum at 
k, (Fs) € K. Then the MER-estimator of the population 
mean is M(F;) = XyB(Fs. 0k) 

MER-estimators with suitable defining functions are 
scale equivariant and do not need a scale estimator. 
MER-estimators are in general consistent estimators of the 
population mean. A proof of the strong consistency of 
MER-estimators of the expectation of a random variable 
is in Hulliger (1991, Chapter 2). 

Problems with nonuniqueness of the minimum or when 
the minimum is not attained on K are easily resolved in 
practice by inspection of the function r(F’5,k). (If there 
are several global minima choose the one with smallest 
tuning constant to obtain more robustness.) The bias part 
of r(F’s,k) involves the slope 8; 5(F's) of the HT-estimator. 
By this term the sensitivity of the HT-estimator is trans- 
ferred to MER-estimators and thus the robustness of 
RHT-estimators is lost again. But if the MER-estimator 
should be consistent for the population mean there is no 
way around a consistent and therefore nonrobust estimator 
in the bias part of the risk estimator. Nevertheless MER- 
estimators are quantitatively less sensitive to outliers and 
more efficient than the HT-estimator if outliers occur (see 
Section 5). 

It is even possible to bound the influence of outliers on 
the MER-estimator for finite samples without loosing its 
(asymptotic) consistency. This is achieved by downweighting 
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the bias part in the estimated risk of the HT-estimator in 
an appropriate way (MER2-estimators, (Hulliger 1991, 
Paragraph 2.4.1)). 

MER-estimators may be more efficient than the HT- 
estimator because their bias is more than compensated 
by the variance reduction due to the downweighting of 
outliers. How much can be gained quantitatively is explored 
in Section 5. 


5. A SMALL SIMULATION STUDY 


Simulations with populations of size N = 128 and with 
samples of size n = 16 are presented here. The sample 
design in Dey and Srivastava (1987) is used (Note that there 
is a factor 2 missing in their formula (2.3)). Dey and 
Srivastava propose to form m > n/2 groups. The group 
totals Lu; xiV = 1, ..., m) must fulfill the inequality 
Yu; xXi/ Luxi > (n — 2)/(n(m — 1)). Thus the group 
totals are allowed only little variability and the groups are 
difficult to form in particular for larger samples (Hulliger 
1991, p. 179). 

The x;(i = 1, ..., N) are independent realisations 
according to a 5%-scale contaminated exponential distri- 
bution with origin at 1, 7.e., (X; — 1) ~ 0.95 Exp(1) + 
0.05 Exp(3), where Exp (@) denotes the exponential distri- 
bution function 1 — exp(—x/6@). The shift +1 is intro- 
duced to lower the probability of negative responses in the 
regression through the origin model with symmetric errors. 

The first response y{}), with acronym GODA, is a 
realization of independent normal variables distributed as 
Y, ~ NV (100x;,x?). This is the model under which the 
HT-estimator is optimal (cf. Godambe 1955). The response 
yi (HTLS) is a realization of independent variables 
distributed as Y¥; ~ W (2x;,x;/4). This is the ideal model 
that yields the HT-estimator as LS-estimator. A third 
response y{?) (HTG) is created by the model Y, ~ 
0.95N (2x;,x;/4) + 0.05N(2x;,9x;/4). The residual 
outliers have 3 times larger scale. The response y{;) (HTE) 
has asymmetric outliers which are not related to the 
x-variable. The bulk of the data (120 observations) stems 
from the distribution Y; ~ WV (2x;,x;/4) of yf?) but 
8 randomly chosen observations stem from Exp(2.5). The 
population y{?) (HMT) stems from a distribution with 
expectation 0.4 + 0.25x; and has a Gamma distribution 
with variance proportional to x*/?. Thus the variable y 
has the distribution proposed in Hansen, Madow and 
Tepping (1983, p.781). Finally a population y{° 
(HMTE) is generated with 120 observations from the same 
distribution as y{>) but with 8 randomly chosen observa- 
tions from the distribution Exp(2). The six populations 
above are chosen to be realistic. They all use the same 
population of x-values (see Figure 1). 


The RHT estimator in the simulation uses 


n(X/ 77) = w(x/,ky) Wau (77.*, meds | rj |), 
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GODA 


HTG 


HMT 


Figure 1. Populations of the Monte-Carlo Study. 


with w(x/,k,) = min(1,k,medy | x/ |/|x/ |) and k, = 
k, = 2. The weighting function w(x; ,k,) corresponds to 
an asymmetric Huber-function Way,,) = min(x/,k,), 
which downweights large x/ only. The scale meds | r/ | is 
the median of the absolute residuals evaluated at the solu- 
tion of the preceding iteration of the iteratively reweighted 
least squares algorithm. The MER-estimator uses the same 
n with tuning constants k,.,k, evaluated at 20 points which 
lie on the diagonal of the range of k, and k,. S-PLUS 
functions for the calculation of the estimators may be 
obtained from the author. 

For each of the populations a set of 400 samples was 
drawn to evaluate the estimators. The obtained precision 
is sufficient to draw conclusions (see the standard errors 
of the efficiencies in Table 1). 

The results are presented in Table 1. The relative bias 
of the RHT-estimator is always larger than the relative bias 
of the MER-estimator. The biases of the two estimators 
have the same sign, except when they are very small. With 
the exception of populations HTE and HMTE the variance 
of the RHT-estimator is larger than the variance of the 
MER-estimator. While the RHT-estimator looses 9% effi- 
ciency at population GODA, where the HT-estimator 
should be optimal, the MER-estimator looses little. With 
population HTLS, where the HT-estimator is the least 
squares estimator, the RHT-estimator looses about 12%. 
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Table 1 
Monte-Carlo simulations with RHT- and MER-estimator 


Populations 

GODA HTLS- HTG HTE HMT HMTE 
MC-mean of HT 6.996 4.531 4.483 2.271 1.068 0.991 
Rel. bias of RHT 0.002 0.001 0.009 0.009 0.006 —0.052 
Rel. bias of MER 0.000 0.001 0.007 0.008 0.002 0.035 
Rel. SE of HT 0.067 0.041 0.044 0.098 0.107 0.170 
Rel. SE of RHT 0.070 0.044 0.040 0.087 0.117 0.144 
Rel. SE of MER 0.068 0.042 0.040 0.091 0.107 0.146 
Eff. of RHT 0.911 0.876 1.110 1.310 0.827 1.234 
Eff. of MER 0.969 0.981 1.158 1.194 0.989 1.284 


MC-SE of eff. RHT 0.020 0.017 0.073 0.009 0.018 0.001 
MC-SE of eff. MER 0.003 0.009 0.037 0.002 0.013 0.002 


NOTE: Relative bias and relative standard error (rel. SE) are biases and standard 
errors divided by the MC-mean of the HT-estimator. Efficiencies (Eff.) 
are MSE of the HT-estimator divided by the MSE of the estimator. 
Estimated standard errors of these Monte-Carlo estimates of efficiency 
are given in the last two lines. 


The efficiency loss of the MER-estimator is once again 
small. Population HTG contains symmetric residual 
outliers. The RHT-estimator gains about 11% (but see the 
error of 7.3%) and the MER-estimator about 16%. Under 
the asymmetric outliers of population HTE the gain of the 
RHT-estimator is 31% while the MER-estimator gains 
19%. If neither the regression through the origin, nor the 
symmetry of errors or the proportionality of their variance 
to the explanatory variable holds, i.e., for population 
HMT, then the RHT-estimator looses 17% compared with 
the HT-estimator while the MER-estimator looses practi- 
cally nothing. If in such a population a few asymmetric 
outliers turn up like in population HMTE then both robust 
estimators gain considerably against the HT-estimator, 
namely 23% and 28% respectively. 

In conclusion from this limited simulation the MER- 
estimator looses little in terms of MSE, compared with the 
HT-estimator, when there are no outliers in the population. 
It gains moderately in populations with symmetric outliers 
and considerably when the outliers are asymmetric. The 
RHT-estimator looses more under ideal situations than the 
MER-estimator. The adaptivity of the MER-estimators 
pays off. 

Extensive simulations with infinite populations in 
Hulliger (1991) confirm these conclusions and show that 
the gains of robust estimators may be very large for skew 
populations with outliers. However the possible efficiency 
gains with robust estimators vanish for large samples since 
then the bias dominates MSE. On the other hand if the 
outliers that turn up in a sample are not representative, 
e.g., if they are uncorrected coding errors, then the robust 
estimators are much more efficient than the HT-estimator 
for all sample sizes. 
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Visitor Sample Surveys 


RONALDO IACHAN and SUZANNE S. KEMP! 


ABSTRACT 


This paper discusses the design of visitor surveys. To illustrate, two recent surveys are described. The first is a survey 
of visitors to National Park Service areas nationwide throughout the year (1992). The second is a survey of recreational 
users of the three-river basin around Pittsburgh, Pennsylvania, during a twelve-month period. Both surveys involved 
sampling in time with temporal as well as spatial stratification. Sampling units had the form of site-period pairs 
for the stage before the final, visitor sampling stage. Random assignment of sample sites to periods permits the 
computation of unbiased estimates for the temporal strata (e.g., monthly and seasonal estimates) as well as estimates 


for strata defined by region and by type of use. 


KEY WORDS: Recreational user; Sampling in time; Site-period. 


1. INTRODUCTION 


Surveys of visitors present unique challenges that are 
rarely discussed in the statistical literature. This paper 
attempts to fill this gap by describing the design and 
emphasizing the common features of two surveys recently 
conducted by the Research Triangle Institute (RTI). We 
hope that the lessons learned in these efforts will be bene- 
ficial to researchers planning similar surveys. 

The first survey was a study of visitors to National Park 
Service (NPS) areas jointly conducted for the National 
Park Service by RTI and HBRS, Inc. This study involved 
a probability sample of park visitors that represented 
visitors to 323 NPS areas nationwide (except Alaska) 
throughout the year (1992). We will refer to NPS areas as 
parks for simplicity while pointing out that the NPS areas 
include locations of historical and cultural parks. The 
main objective of the NPS study was to assess the visitors’ 
experiences and problems with particular attention to 
those related to aircraft overflights (e.g., noise and other 
possible annoyances). A variety of data were also collected 
in a mail survey for a subsample of selected visitors. 

The second survey was a study of recreational users of the 
Pittsburgh-area three-river basin along the Monongahela, 
Allegheny and Ohio Rivers in 1992 (or more precisely, 
between February 1992 and January 1993). This survey 
was jointly conducted for the Ohio River Valley Sanitation 
Commission by RTI and Terrestrial Environmental 
Specialists. The study area included a 40-mile segment of 
the Ohio River, a 24-mile segment of Monongahela River 
and a 7-mile segment of the Allegheny River. The primary 
objective of the Three-River Study was to construct a 
baseline profile of recreationists in the area and to model 
the economic value they assign to various activities. Three 
basic types of recreational activities were distinguished: 
boating, fishing and park use. 


The Three-River Study is the most comprehensive of 
a series of studies conducted by RTI to assess environmental 
impact in a number of states. These studies estimate possible 
reductions in economic or recreational value assigned by 
actual and potential recreational users to areas that have 
been or might be affected. While a wider survey of potential 
users of such areas may consider a telephone sample design, 
a visitor intercept survey design is found necessary to capture 
users at a point in time close to actual use. 

A discussion of design issues in visitor surveys such as 
these has been recently provided in Kalton (1991) including 
issues related to sampling in time and space that are crucial 
in our framework. In its simplest form, a prototype, 
two-stage sample design for a visitor survey considers site- 
period pairs as primary sampling units (PSUs) from which 
visitors are selected in the second stage. Examples include 
exit polls (see, for example, Levy 1983), shopping mall 
intercept surveys (see, for example Sudman 1980) and 
other transportation and traffic surveys (Gough and 
Ghangurde 1977; Kish, Lovejoy and Rackow 1961). 
Among the design issues salient in visitor surveys, the 
following general problems may be singled out: 


— It is desirable to select with greater probabilities those 
site-periods with larger numbers of visitors; stratification 
and PPS selection are then effective design features. 


— Data collection arguments are key for the specification of 
the period length and of sampling rates within site-periods; 
e.g., trade-offs occur between the potential for the field 
staff to be too busy (short periods, high sampling rates) 
or not busy enough (long periods, low sampling rates). 


— Analytic objectives as well as efficiency suggest temporal 
stratification dimensions as season, month, weekend 
versus weekday, and even time-of-day; e.g., the need for 
seasonal estimates suggests the use of seasons or months 
as strata for the selection of periods. 
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The two surveys discussed in this paper share the primary 
objective of characterizing the visitor population in the 
area or the nation over an entire year. They differ, how- 
ever, in the priority estimates that lead to the basic design 
features in each case. In the Three-River Study but not in 
the NPS study, reasonably precise monthly estimates 
needed to be computed. For the former study, then, the 
temporal sampling units - days - were stratified into 
months. Spatial stratification of the Three-River Study 
sites was geographic and by recreation type (boating, 
fishing or park sites). 

For the NPS study, primary stratification was by park 
type. Some park areas needed to be included with certainty 
into the sample to satisfy legislative requirements. In these 
and other selected parks, it was further desired to compute 
park-specific estimates. In these park areas, labelled inten- 
sive parks, we then decided to select relatively more site- 
periods. The initial design optimization problem was how 
to allocate the sample size to the sampling stages, /.e., to 
decide how many parks and how many site-periods per 
park should be selected. Section 3 discusses a solution for 
this problem which is a function of the intracluster corre- 
lations within parks and within periods. The design opti- 
mization for the NPS survey also applied to the temporal 
and spatial strata at the intermediate sampling stages, 
between park areas at the first stage and site-periods at the 
penultimate stage (keeping in mind that visitors are selected 
at the final stage). 

These two surveys illustrate issues such as temporal 
stratification, the choice of appropriate sampling units, 
and random assignment of spatial units to temporal units. 
Section 2 outlines the common aspects of the two studies 
as well as their basic differences. Sections 3 and 4 describe 
the design of the NPS Visitor Survey and of the Three- 
River Study, respectively. Section 5 discusses the weighting 
procedures used for the surveys. A brief overview and 
some conclusions are presented in Section 6. 


2. OVERVIEW OF SAMPLE DESIGNS: 
PARALLELS 
AND CONTRASTS 


For both surveys, the ultimate visitor samples were 
selected via intercept sampling as visitors left the sample 
locations at the selected time periods. Exit interviews were 
necessary to reflect their attitudes immediately following 
their recreational experiences. Also, in both studies, 
visitors were selected from sample site-period pairs. The 
use of site-period pairs as sampling units dates back to 
Kish, Lovejoy and Rackow (1961). This sampling unit 
definition permits the selection of visitors according to a 
data collection schedule that specifies which sites will be 
covered at which points in time. Unlike the Three-River 
Study, the selection of site periods was not the first stage 
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for the NPS Study. The primary sampling units (PSUs) 
for this study were NPS areas, or parks. The NPS survey 
involved several stages of selection described in the next 
section. 

Additionally, both studies used temporal frames of days, 
and eligible data collection periods within days, to permit 
inferences about the entire year. The designs included the 
selection of time periods so that each eligible period has 
a known, positive probability of selection. Although both 
studies involved temporal frames, the structure of the 
frames and selection of days for each study were quite 
different. 

For example, the sample for the Three-River Study was 
selected as twelve independent monthly samples. Each 
monthly sample has essentially the same, stratified random 
sampling design but a different sample allocation and 
different sample sizes were used in different months. This 
design took into account seasonal variations in recreational 
patterns, and enabled estimation for each month and 
stratum (e.g., by type: boating, fishing or park). Both 
spatial and temporal frames were allowed to vary from 
month to month. The stratification and allocation for this 
sample are discussed in Section 4. 

In contrast, the temporal frame for the NPS visitor 
survey first considered two-month blocks for each sample 
park (PSU). The use of two-month periods as (second- 
stage) sampling units in time efficiently met the survey 
objectives for two basic reasons. First, it allows the effec- 
tive (geographic) concentration of staffing resources and 
staggered data collection throughout the year. Second, this 
choice of period permitted capturing seasonal fluctuations 
in park visitation across the park system, resulting from 
some parks having relatively higher visitation in the spring, 
others in the fall months, and so on. 

One two-month block was selected for each sample 
park so that data collection could be effectively concen- 
trated in time. Then, at the next stage of temporal selection, 
days were selected from within the two-month block for 
each sample park. Like the parks themselves at the first 
stage, these two-month blocks were selected with proba- 
bilities proportional to size (PPS), with the size measure 
being the aggregate visitation. 

The sample sizes and allocation to the several sampling 
stages were carefully balanced to minimize clustering 
effects associated to clusters in time and space. For the 
Three-River Study, this clustering occurs at the first stage 
of selection where sampling units are sites and time periods. 
The allocation also considered the varying sample sizes 
used in successive, independent monthly samples. For the 
NPS survey, clusters in time were a result of the two- 
month blocks and sample days periods selected at different 
stages. Spatial clusters resulted from the use of parks and 
park exits as sampling units for this survey. 

The next section describes in more detail the design of 
the NPS survey. 
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3. NPS SURVEY SAMPLE DESIGN 


The design of NPS visitor survey capitalized on auxiliary 
information of various kinds and sources: 


- Information obtained in previous studies (e.g., park 
rankings based on noise exposure and NPS staff 
classifications). 


- Information available in NPS data bases (e.g., park 
visitation data by month). 


- Information collected from NPS staff specifically for 
design purposes (e.g., an inventory of the park exits for 
each sample park and number of vehicles leaving each 
of the exits). 


The next subsections describe the various stages of 
selection for the visitor intercept survey. This survey com- 
ponent will also be designated the frontcountry survey to 
distinguish it from a survey of backcountry users that was 
conducted in tandem in the sample parks. Subsection 3.5 
describes the backcountry survey as well as a mail survey 
administered to a subsample of frontcountry respondents 
and to sample backcountry users. 

The fourth-stage selection for the mail survey involved 
additional stages (and phases) of selection. For the visitor 
intercept survey, groups (i.e., vehicles) were selected as an 
ultimate cluster: all persons in a sample group were solicited 
for an interview. For the mail survey, groups were sub- 
sampled, and one person was subsampled from each 
subsampled group. 


3.1 Frame Construction and First-Stage Sampling 


We constructed a sampling frame for the selection of 
parks by compiling NPS information on park visitation 
(monthly and annual) and on noise exposure, information 
that was used in the sample design in two distinct ways: 
for stratification and for assigning size measures. The 
latter information was based on two different sources: 
(a) a previous NPS study which ranked parks according 
to potential exposure, and (b) a classification of parks 
performed independently by NPS staff (park super- 
intendents, regional staff efc.). 

In consultation with NPS, RTI combined these two classi- 
fications for noise exposure to construct eleven strata. The 
stratification partitioned parks into categories — very high, 
high, low and very low. Strata were divided into two substrata 
using the rankings in the stratum. (Note that the ‘‘medium”’ 
stratum was not subdivided due to its small park count.) 

In addition to noise exposure, these strata incorporate 
three classes of parks that deserve separate treatment: 
(1) urban and suburban park areas, (2) parks with missing 
data on visitation (needed for PPS selection), and (3) parks 
whose elongated shapes present unique problems of access 
and reduce the meaning of prior exposure assessments. 
These classes were sampled at a much lower rate than the 
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other strata; the lowest sampling rate is in the urban 
stratum (1 in 79). 

The certainty stratum included the seven parks that 
were mandated by legislation to be included in the study. 
In addition, it included those parks whose aggregate 
(annual) visitation rates were so large as to ensure selection 
into the first-stage sample. The 39 sample parks are listed 
in Exhibit 1; a 40th selected park (Grand Teton) was 
dropped from the sample for political reasons. 

Design optimization calculations led to a first stage sample 
size of about 40 sample parks, yielding a total of 40S site- 
periods (or exit-days in this case) selected across intensive 
and non-intensive parks. As described in Section 3.3, 15 exit- 
days were selected in each of the three intensive parks, and 
10 exit-days were selected in each of the 36 non-intensive 
parks in the sample. An accurate optimization would require 
variance components for the between-park and within-park 
variances. These variance components were approximated 
using data from a previous study as well as monthly visitation 
data available for all NPS areas. 


3.2 Selection of Two-Month Blocks 


While the selection of two-month blocks was with prob- 
ability proportional to size (PPS), practical requirements 
were also taken into consideration. 

First, training local park staff immediately before data 
collection was desired. Geographic clusters of parks were 
formed so that trainers could visit parks in one cluster in 
one trip. Specifically, the 39 sample parks were grouped 
into 14 such clusters. This requirement led to the selection 
of a two-month period for each park cluster. Thus, fourteen 
two-month blocks were selected, one for each cluster. The 
size measure used for each selection was the aggregate 
visitation over the parks in the cluster. 

Exhibit 1 shows the sample parks in each cluster, and 
the sample two-month block selected for the cluster (i.e. 
for every park in the cluster). Three strata - groups of 
clusters and hence of parks - were formed for the selection 
of two-month blocks: 


(a) In the ‘‘very-high summer”’ stratum-1 (with 3 clusters), 
the frame of two-month blocks contained only the 
summer-peak period, July-August, which was then 
selected with certainty for these clusters; 

(b) In the ‘‘high-summer’’ stratum-2 (with 7 clusters), the 
frame of two-month blocks contained 11 overlapping 
two-month blocks; 

(c) In the ‘‘low-summer’’ stratum-3 (with 4 clusters), the 
frame of two-month blocks contained 5 non-overlapping 
two-month blocks. 


The rationale for this temporal stratification was two-fold: 
it ensured that the data collection was spread throughout 
the year, and it distinguished parks where a vast majority 
of the visitation occurs during the summer from those with 
a more uniform visitation pattern. 
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Exhibit 1 
Park Clusters, Second-stage Strata and Selected 
Two-month Blocks 


Selected Period 
Stratum 1: Very High Summer* ; 


Cluster 2: Mount Rainier, N. Cascades, July-August 
Olympic 

Cluster 4: Glacier, Yellowstone July-August 

Cluster 10: Sleeping Bear, Perry’s Victory July-August 


Stratum 2: High Summer 


Cluster 3: Lassen, Yosemite, Kings 
Canyon/Sequoia 


August-September 


Cluster 5: Dinosaur, Rocky Mtn., June-July 
Mt. Rushmore 

Cluster 6: Glen Canyon, Grand Canyon, June-July 
Walnut Canyon 

Cluster 8: Bandelier, Lake Meredith May-June 


Cluster 11: Cape Cod, Delaware Gap, August-September 


Gettysburg 

Cluster 12: Shenandoah, Fredericksburg, June-July 
Assateague 

Cluster 13: Great Smoky, Cape Hatteras, June-July 
Fort Sumter 

Stratum 3: Low Summer 

Cluster 1: Haleakala, Hawaii Volcanoes March-April 

Cluster 7: Lake Mead, Saguaro,CasaGrande March-April 


Cluster 9: Hot Springs, Wilson’s Creek, October-November 
Buffalo 
Cluster 14: Cumberland Isd., Canaveral, 


Everglades, Gulf Island 


March-April 


* Certainty selection of July-August for each cluster in this stratum. 


We selected two-month blocks with different procedures 
in the two strata as described below. 


(1) For park clusters in the former (high-summer) stratum, 
eleven overlapping periods were included in the tem- 
poral frame; January-February, February-March, 
..., November-December. One such period was then 
selected with probability proportional to size (PPS). 
Note that for this stratum each month was included in 
two frame periods except for January and December. 
The probability of selection for these two winter 
months was thus reduced even further (beyond the 
already small probability assigned to the winter periods 
with the PPS procedure). 


(2) For park clusters in the latter (low-summer) stratum, 
five non-overlapping two-month periods constituted 
the temporal frame: January-February, March-April, 
May-June, September-October, November-December. 
Note that the two-month summer period, July-August, 
was excluded from the frame for these parks to ensure 
the selection of other, non-summer months. For each 
cluster in this stratum, one of these five two-month 
periods was selected with PPS. 


lachan and Kemp: Visitor Sample Surveys 


3.3. Third-stage Sampling 


Sampling units at the third stage were exit-day pairs. 
The third-stage sampling is easier to envisage as the combi- 
nation of two independent selections: (a) selecting days 
from the temporal window (2-month block) drawn at the 
second stage for the park, and (b) selecting exits from a 
list of exits specified for the park. A final step consisted 
of the random assignment of sample days to sample exits. 

The sample of days was stratified by weekdays versus 
weekend days (including major holidays). Sample days 
were selected with equal probabilities within each of these 
two strata. The sample of exits was selected with proba- 
bilities proportional to size (PPS). The size measure 
assigned to each exit was the relative use of the exit among 
all the exits listed for the selected park. This usage measure 
was derived with the aid of local park staff. 

The third-stage sample design distinguished two groups 
of parks designated as intensive (3 parks: Grand Canyon 
and the two Hawaii parks) and non-intensive (36 remaining 
parks in the sample). Sample sizes in non-intensive parks 
were 10 exits and 10 days, and hence 10 exit-day pairs. In 
the three intensive parks, 15 exits and 15 days were selected. 
Equal allocation to weekend/weekday strata was used in 
non-intensive parks: 5 weekend days and 5 weekdays were 
independently selected in each of these (36) sample parks. 
The allocation was approximately equal in intensive parks 
with the selection of 7 weekend days and 8 weekdays. 


3.4 Fourth-stage Sampling 


At the fourth stage, park visitors were intercepted in 
the selected exit-days. A systematic random sample of 
visitors or (visitor groups) exiting the park was selected 
with a fixed sampling interval for each selected third-stage 
unit (exit-day). The interval was allowed to vary from day 
to day to capitalize on the experience of previous days and 
on the variability in visitation across days and exits. Each 
visitor found in the selected eligible groups was screened 
for eligibility, and interviewed if eligible (adult visitor). 


3.5 Backcountry and Mail Surveys 


The same sample of parks (PSUs) was used for a mail 
survey of frontcountry and backcountry users in the two- 
month period selected for the park. The sample for the 
backcountry survey was restricted to the subset of sample 
parks with some backcountry use. Within each such sample 
park, the (third-stage) sampling frame for this survey 
component was based on backcountry permits issued 
during the data collection time window (two-month block) 
established for the park. The ultimate sample of back- 
country users was then selected with equal probabilities 
from permit lists provided by park staff. A subsample of 
visitors selected in the intercept survey were also selected 
for the mail survey. 
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The mail survey sample was based on the same third- 
stage sample of exit-days selected for the intercept survey. 
For each selected exit-day, a fixed number of groups was 
subsampled from groups responding to the intercept 
survey (this number was 15 in intensive parks and 10 in 
non-intensive parks). A further stage of subsampling was 
that of one person from within the respondents in each 
group subsampled. This subsampling of groups and 
persons was with stratified random sampling to control 
the demographic composition of the final sample. 


4. SAMPLE DESIGN FOR THREE-RIVER SURVEY 


4.1 Frame Construction and Stratification 


First, RTI and Terrestrial Environment Specialists 
(TES) constructed a sampling frame based on an inventory 
of all sites in the Three-River Study area. Then a stratified 
multistage sample was selected independently for each of 
the twelve months of the study. First-stage sampling units 
were site-periods, and second-stage units were individuals 
engaged in recreation in the selected site-periods. Temporal 
and spatial stratification were used for the first-stage 
sampling of time periods and sites. 

Primary stratification along the temporal dimension 
was by month, and primary stratification of sites was by 
use type: sites (access points) were classified as boating, 
fishing, or parks. Each primary stratum was divided into 
six geographic areas, or pools, defined as the river areas 
between locks. The fishing stratum was further substratified 
by the presumed use intensity as low or high use: high-use 
sites are those below locks and dams. 

Each monthly sample of site periods was selected with 
stratified random sampling. Advantages of selecting 
independent monthly samples included: monthly and 
seasonal estimates can be computed, and some design 
features may be changed from month to month. 

For example, this design permitted altering the spatial 
frame from one month to the next with the addition or 
deletion of sites. In particular, the boating stratum for the 
winter months (November through April) was restricted 
to boat ramps open that season. Further, several fishing 
sites included in the frame for the first few months were 
found inaccessible and deleted from the frame for the 
subsequent monthly samples. 

Other design features that changed in successive months 
include: the second-stage sampling rates for selecting 
eligible users, and the data collection windows used in the 
morning, afternoon and evening periods for sites of each 
type. Varying data collection windows were used in differ- 
ent months of the study. These periods were defined using 
sunrise and sunset information as well as expected patterns 
of use of the various types. The winter months (November- 
April) included two periods per day while the summer 
months (May-October) included three periods per day. 
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The temporal (sub)stratification of each monthly sample 
was by weekend days versus weekdays. It is worth noting 
that the weekend stratum also included major holidays. 


4.2 Sample Selection 


As noted above, we selected an independent first-stage 
sample of site-periods for each of the 12 months of the 
study. Each monthly sample had two components: (a) a 
stratified random sample of ‘‘n’’ sites, and (b) a stratified 
random sample of ‘‘n’’ periods. 

Following selection of the sample periods for each 
month, the sample sites were randomly assigned to the 
selected periods. The assignment of sites to periods was 
entirely at random for the months of February through 
June but was modified in subsequent months. From July 
on, a sample of time periods was independently selected 
for each type stratum with the random assignment taking 
place within stratum. The allocation of the sample time 
periods (e.g., the number of morning periods and the 
number of evening periods included in the sample) varied 
from stratum to stratum. With this more flexible method, 
relatively more fishing sites could be assigned to morning 
period and more boating sites to afternoon periods, for 
example. Exhibit 2 shows the sample sizes - sites and 
periods - used in random assignment each month. 


Exhibit 2 


Sample Sizes Used in Random Assignment of Sites to Periods 
for each Monthly Sample of Three-River Study 
Sample Size 


Month Overall* MM ———— Parks Marinas 
Boating Fishing 


Regattas 


Boating Marinas 


1 8 12 4 

2) 20 

3 28 

4 28 

5 36 

6 36 

7 17 22 6 12 
8 10 pip) 6 6 6+ 6+ 
9 10 14 6 6 
10 10 12 4 

11 8 12 4 

12 8 12 4 


* 


For these (5) months, the assignment took place for the entire collection of 
sample sites and periods (NOT blocked by site type). For the remaining months 
the assignment was within each type stratum, a process which involved the 
selection of independent samples of time periods for each type. 

In August, the assignment of regatta sites to periods was performed first, 
separately. Following the assignment, the sample site-periods associated with 
regattas were shifted either to the boating or to the marina strata depending on 
the site type. 


+ 


Second-stage sampling rates were specified for each of 
the three primary strata prior to each month of data 
collection. These stratum-specific rates were determined 
based on the experience of the previous months, and were 
distributed to the field interviewers along with the month’s 
data collection schedule. Individuals were selected with 
systematic random sampling within each site-period. 
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4.3. Marina Survey and Special Events 


A marina survey was conducted in the months of June 
to September. A sampling frame of 48 marinas was based 
on the TES inventory that was updated in late May 1992. 

The sampling method for the phase-in month of June 
differed slightly from that used for the subsequent monthly 
samples for marina sites. The June sample was a supple- 
ment of 12 marina sites coupled with a sample of 12 days. 
The marina samples for the months of July to September 
were selected considering the marina frame as a fourth 
stratum. The selection procedures were then similar to 
those in the other three type strata; specifically, (a)a 
sample of ‘‘n’’ marina sites was selected, (b) a sample of 
‘*n’’ time periods was selected, and (c) the sample sites 
were randomly assigned to the selected periods. 

It is worth pointing out that some of the marina sites 
were also included in the boating stratum. In such cases, 
two distinct frame units were created for the same site. This 
situation also arose for some sites used for both boating 
and fishing, and such sites were included in both strata. 

In addition to the marina survey, we identified special 
events taking place in the study area over the 12-month 
study period. Most of these events were handled in a way 
similar to weekends and holidays by assigning them to a 
stratum to be oversampled. A special category of interest 
was comprised of the regattas occurring in the summer 
months. For two monthly samples (July and August), we 
identified the regatta dates as well as the sites affected by 
each regatta. We then constructed a separate (fifth) stratum 
to include these site-periods. The first-stage sample alloca- 
tion to the regatta stratum reflected the oversampling 
desired for this stratum. As shown in Exhibit 2, the 
sampling procedure used to select site-periods (first-stage 
units) from the regatta stratum also differed to that used 
in the other four strata. Sample site-periods were directly 
selected in one step from the subset of site-periods in the 
stratum, /.e., no random assignment was needed. 


5. SURVEY WEIGHTING 


5.1 NPS Survey Weighting 


Sampling weights were first computed for each of the 
first three stages of selection. The first-stage sampling 
weight for each sample park was the reciprocal of the selec- 
tion probability for the park. The second-stage sampling 
weight for each sample two-month block was similarly 
computed. The sum of the first-stage sampling weights 
overall (or in a stratum) was the number of parks in the 
frame (or in a stratum). 

Third-stage weights for sample exit-days were the product 
of two factors associated with the selection of days and 
exits. Note that for each selected park and two-month block, 
the sum of the former set of weights in a temporal stratum 


lachan and Kemp: Visitor Sample Surveys 


(weekend vs. weekdays) is the number of days in the stratum, 
and the sum of the latter set of weights is the number of 
exits listed in the park. These weights were adjusted for 
nonresponse which arose at the third stage because in a few 
parks, data collection did not take place in some selected 
exit-days. In a given park with this data collection short- 
coming, the sum of the adjusted third-stage weights over 
the active exit-days was made equal to the sum of the 
sampling weights over all selected exit-days in the park. 

Fourth-stage weights were computed at a group-level 
and at a person-level. Group-level weights are assigned to 
all participating groups in a sample exit-day, and have the 
same value for the groups in the same exit-day. Similarly, 
person-level weights are assigned to all persons intercepted 
in a sample exit-day. The fourth-stage sampling weights 
were computed as the reciprocal of the sampling rate 
specified for the sample exit-day. These weights were then 
adjusted for group and person-level nonresponse. 

The mail survey sample was based on the same third- 
stage sample of exit-days selected for the intercept survey. 
For each selected exit-day in non-intensive parks, 10 groups 
were first selected with equal probabilities from among the 
participating groups; then, one person was subsampled 
from all intercept survey respondents in each selected 
group. A similar procedure was used in intensive parks 
with the exception that the number of selected groups per 
exit-day was 15 rather than 10. 

The sampling weight for each mail survey record is the 
product of WTB = number of intercept respondents in 
the group, and WTA = number of participating 
groups/10 [non-intensive parks]. For intensive parks, the 
denominator of WTA is 15 rather than 10. 

These weights were adjusted for mail survey nonresponse 
using exit-days (within park) as weighting classes. Thus, the 
sum of the adjusted weights for all respondents coming from 
the same exit-day is the same as the sum of the (unadjusted) 
weights for all persons intercepted in the exit-day. 

For the backcountry survey, the sampling frame for each 
eligible park (in the subset of sample parks with backcountry 
use) was based on lists of permits issued during the data 
collection period: eligible permits were associated with exit 
dates in this period. A sample of 5 groups per day was selected 
within each park from the set of permits linked to that day. 
One person was subsampled from each group. The weight 
computation parallels that for the mail survey with 


WTBACK = BACKA*BACKB, 


where for each day: 
BACKA = (number of groups linked to the day)/5 
BACKB = number of persons in group. 


Analysis weights for the backcountry survey resulted 
from nonresponse adjustments made using days as weighting 
classes. 
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5.2 Three-River Survey Weighting 


As each monthly sample of site-period units was 
selected, we computed sampling weights that reflected the 
selection probabilities for the site-period pairs. Initial 
weights were the product of two sets of weights computed 
for each monthly sample: (a) weights assigned to each site 
in the stratified random sample of sites of the given type 
(boating, fishing, marina and park), and (b) weights 
assigned to each period in the stratified random sample 
of periods. These weights were then inflated to take into 
account the random assignment of sample sites to periods 
(or vice-versa). Thus for each month, the sum of the site- 
period weights was equal to the number of site-period 
combinations in the frame. 

The weight adjustment process started with the sampling 
weights associated with the selected site-days. An initial 
adjustment was made to the first-stage weights to account 
for site-periods that were found ineligible or that had 
missing sampling forms (e.g., not sent by field staff). For 
this first-stage adjustment, we used the type-by-month 
strata as weighting classes. That is, the sum of the adjusted 
weights over the reduced set was made equal to the sum 
of the unadjusted weights over the entire sample within 
the type-by-month class. A final adjustment was made at 
the respondent-level to reflect (a) the systematic sampling 
interval used within site-periods, and (b) the survey non- 
response at the individual level. 

As part of the weight check procedures, we computed 
the sum of the final analysis weights over the entire file, 
and also by month and by type of site. The weight sum 
should approximate the estimated total number of recrea- 
tional users leaving the inventory sites during the data 
collection time window for each month and type of site. 


6. CONCLUSION 


This paper described the design of two surveys of 
recreational users that share a number of useful features. 
The sample designs include sampling in time as well as in 
space; site-periods are selected at the stage prior to sampling 
visitors. A spatial frame is then constructed side by side 
with a temporal frame. After sites are selected from the 
former and periods are selected from the latter frame, 
sample sites are randomly assigned to sample periods (or 
conversely). Sampling weights need to take into account 
this additional step of randomization. The findings of the 
NPS visitor survey are described in the study final report 
(National Park Service 1994). This analysis included a 
variety of regression models that investigate the impact of 
hearing and seeing aircraft flying over NPS areas. 

Temporally, both studies represent periods throughout 
the year, that is, a user will have a positive probability of 
selection for any time of the year. Both studies also include 
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temporal stratification to reflect patterns of use and 
increase sampling efficiency. 

Spatially, while the sample for the National Park Service 
study was a national sample of visitors to NPS areas, the 
sample for the Three-River Study was more restricted in 
spatial scope. Both studies, however, distinguished users 
of different types. For the NPS study, backcountry and 
frontcountry users were selected in two separate (third-stage) 
strata at the point where the two components branch out 
with the selection of permits for the backcountry survey. 
For the Three-River Study, visitors were classified by primary 
fishing, boating, or park use, and sites were stratified in 
a similar way. This design permitted the computation of 
precise estimates by type and by season. 

The ultimate sampling unit for a survey of recreational 
users is the specific visit; thus, visitors may have multiple 
chances of inclusion in the sample to the extent that they 
use the target areas multiple times. It is worth noting that 
this structure is consistent with the objectives of such 
surveys. 

Sampling weights accounted for the selection of time 
and space units at each stage and also for the random 
assignment step. The samples were designed to minimize 
the effects of unequal weighting on survey variances. The 
potential for severe unequal weighting effects was consid- 
ered in combining different survey components. Examples 
in the two surveys include combining: 


(a) Backcountry and frontcountry components of NPS 
mail survey, and 

(b) Fishing, boating and park users in the Three-River 
Study. 


Some disadvantages of this type of study design should 
also be pointed out. While sampling in time and random 
assignment introduce an element of statistical rigor and 
extend the range of valid statistical inferences, the method- 
ology may be disrupted if field interviewers change the 
date assigned for a sample location because access may be 
difficult in the specified period or for other reasons. 
Noteworthy examples in the NPS study included hurri- 
canes, park closed due to fugitives from justice, space 
shuttle launches, and severe snowstorms. While some of 
these occurrences may be minimized with the temporal 
stratification and allocation, others are clearly beyond the 
control of the statistician. However, the sampling statis- 
tician should be involved in interviewer training to stress 
that modifications in the sample schedule should be avoided 
at all costs, and should monitor any changes that do occur. 
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In This Issue 


This issue of Survey Methodology contains papers on a variety of topics. The first paper, by 
Davis and Scott, discusses the impact that interviewer effects may have on comparisons between 
domain means. Using a components of variance model, it is shown theoretically that the impact 
depends on the distribution of each interviewer’s case load between the domains and on the domain- 
interviewer interaction. The model is applied to data from a health survey to estimate the magnitude 
of interviewer effects for comparisons between sexes and between ethnic groups. It was found that 
in some cases the domain-specific interviewer effects have a large impact on the accuracy of between 
domain comparisons. 

Rivest and Hurtubise examine the usefulness of the Winsorized mean as an estimator of the 
mean of a population that has a distribution skewed to the right. A Winsorized mean is obtained 
by replacing all observations greater than a given threshold value R by this same value R, before 
the mean is calculated. The authors suggest a simple algorithm for calculating R that minimizes 
the squared error of the estimator. They apply this method to several sample sizes and various 
sample designs, including stratified sampling and sampling with probabilities proportional to size. 
They derive direct approximations of the effectiveness of the Winsorized mean. They conclude 
their article with a Monte Carlo simulation to compare various estimators that reduce the impact 
of extreme values. 

Kish, Frankel and Verma examine the possible incidence and the importance of the design 
effect (deft) on a set of interrelated statistics. On the basis of 14 surveys conducted in six countries, 
the authors present an empirical approach relating the design effect of analytical statistics, 
deft(p; — p;), to the design effects of separate statistics, deft(p;) and deft(p;), for two of the 
many categories of the same variable. The proposed approximation must be checked constantly. 
However, it appears to be widely applicable to the data studied, and it is clearly preferable to the 
hypotheses put forward thus far on deft(p; — pj). 

Dupont discusses the estimation of a total from a two-stage sample where auxiliary information 
is present. First three regression estimators are presented, each making different use of the auxiliary 
information. Then four calibration estimators are proposed, each corresponding to a specific 
strategy for using auxiliary information. Dupont then shows that the calibration strategies can be 
association with regression modelling. This article also discusses variance estimation for the seven 
estimators presented, the choice of the estimator where there is nonresponse, and the a priori or 
a posteriori use of auxiliary information. 

Spisak discusses the use of statistical process control to assure the quality of a frame constructed 
by a continuous process and used for a survey repeated periodically. The frame sizes constitute 
a time series for which the appropriate model must be identified in order to estimate the process 
variance needed for construction of the control charts. The author uses the data from the United 
States Unemployment Insurance Benefits Quality Control program to illustrate the method. 

Binder and Kovacevic show how the estimating equations approach may be used to construct 
variance estimation procedures that are appropriate when the data come from a survey with a 
complex design. The approach is most useful when the quantity to be estimated is a complicated 
non-linear function of the survey population values, as is the case with many common measures 
of income inequality. Details of the proposed approach are worked out for a number of complex 
income distribution statistics including the Gini Coefficient, the Lorenz Curve Ordinate, the 
Quantile Share, and the Low Income Measure. A numerical example is given using data from the 
Canadian Survey of Consumer Finance. 
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In This Issue 


Ernst and Ikeda present a reduced-size algorithm for maximizing the retention of selected primary 
sampling units when a new sample (i.e., with a new stratification and allocation) is selected for 
a repeated survey. First, the transportation procedure developed by Causey, Cox and Ernst (1985) 
is described. It provides optimal retention of PSU’s but the resulting transportation problem may 
be too large to solve in practice. The authors then expose their algorithm which is an approximation 
of the previous method but has the advantage of being of smaller size and thus possible to use in 
many practical situations. Finally, an application of the algorithm to the Survey of Income and 
Program Participation is presented. 

Shrestha and Preston evaluate the consistency of the Census data with the Vital Registration 
data for the older Americans. First, the data used in the study and their sources of errors are 
described. Then the authors present the methodology used to evaluate the quality of the old-age 
statistics and explain how one should interpret the results of the application of that methodology. 
Finally, results from the application of the methodology to data from 1970 to 1990 are presented. 

Dillman, Clark and Sinclair compare different mailout and follow-up strategies with respect 
to their impact on the response rates for the U.S. Census. The comparison of the strategies includes 
the use of a factorial design and a sample of 50,000 housing units. The results are analyzed through 
multiple pairwise comparisons of treatment means and logistic regression. 

Forster and Snow evaluate the use of hand-held computers to conduct demographic surveys 
in developing countries. A data collection test was conducted for comparing the use of paper and 
computerized questionnaires with the Adult Mortality Survey of people living on the Kenyan coast. 
The results show that the use of hand-held computers can reduce the data processing time, improve 
the quality of the data as well as reduce survey costs on the long term. 


The Editor 
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The Effect of Interviewer Variance on Domain Comparisons 


PETER DAVIS and ALASTAIR SCOTT! 


ABSTRACT 


In this paper we explore the effect of interviewer variability on the precision of estimated contrasts between domain 
means. In the first part we develop a correlated components of variance model to identify the factors that determine 
the size of the effect. This has implications for sample design and for interviewer training. In the second part we 
report on an empirical study using data from a large multi-stage survey on dental health. Gender of respondent 
and ethnic affiliation are used to establish two sets of domains for the comparisons. Overall interviewer and cluster 
effects make little difference to the variance of male/female comparisons, but there is noticeable increase in the 
variance of some contrasts between the two ethnic groupings used in this study. Indeed, the impact of interviewer 
effects for the ethnic comparision is two or three times higher than it is for gender contrasts. These findings have 
particular relevance for health surveys where it is common to use a small cadre of highly-trained interviewers. 


KEY WORDS: Interviewer variance; Domain comparisons; Design effect. 


1. INTRODUCTION 


Surveys requiring a high degree of specialist training for 
interviewers, such as many health studies, are often forced 
to use a small number of highly-trained interviewers. 
There has been a substantial amount of work done on 
estimating the impact of interviewer variability on simple 
statistics such as means and proportions, and it is well- 
known that the use of a small number of interviewers, each 
having a high case load, can lead to a relatively large con- 
tribution to the total error. Comprehensive summaries of 
the literature are given in Groves (1989, chap. 8) and Lessler 
and Kalsbeek (1992, §11.3). However, most medical and 
social surveys are primarily interested in more complex 
questions such as comparisons between sub-groups or 
estimating the effect of a factor on disease outcome. There 
is a widespread belief that the effect of interviewer vari- 
ability is much smaller here, and that the effect of a small 
number of interviewers is relatively harmless. Following 
the pioneering work of Kish and Frankel (1974), there has 
been a great deal of theoretical and empirical work on the 
effects of clustering on fitting multiple regression models 
or log-linear models for categorical data. Good accounts 
of the literature are given in Skinner ef a/. (1989) and Rao 
and Thomas (1988). There has been some empirical work 
on the conceptually simpler, yet practically important, 
problem of comparing sub-group means (see Kish 1987 
and Skinner 1989 for example) but relatively little theo- 
retical development. 

In this paper we concentrate on comparisons between 
subgroups (or domains). We first look at theoretical 
aspects via a straightforward components of variance 
model. The theory suggests that the impact of interviewer 


variability depends on two things, the distribution of each 
interviewer’s case load between the domains and the 
domain-interviewer interaction. Then we apply the theory 
to data from a reasonably typical health survey, using two 
sets of domains defined by the sex and ethnic background 
of the respondent. Unfortunately the study was not 
designed a priori to estimate interviewer effects (most 
importantly, interviewers were not deployed at random) 
so the results should be regarded as suggestive rather than 
definitive. However, they are sufficiently disturbing to 
indicate that the problem warrants further study. The 
results from the ethnic comparisons, in particular, suggest 
that there are cases when we should be concerned about 
using a small number of interviewers even when compar- 
isons, rather than simple means or proportions, are the 
main concern of the analysis. 


2. THEORY 


For simplicity we start with the special case of a two- 
stage self-weighting design. This is sufficiently complex to 
illustrate the central ideas, but simple enough to avoid 
being swamped with extraneous detail. Following Collins 
and Butcher (1982), we want to address the problems of 
interviewer variance and clustering together. A simple 
correlated response model appropriate for observations 
drawn according to such a design is 


Yipr nan hia by + iprs (1) 


where / denotes the interviewer, p the primary sampling 
unit (PSU) and r the individual respondent. Here the 
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mean, p, is fixed constant and the remaining components, 
a;, b, and @;,,, are assumed to be independent random 
variables with variances 07, 02 and o” respectively. Such 
models have been used widely in theoretical studies of 
response variance. See Prasad and Rao (1990) for a recent 
example. For references to earlier work, see the compre- 
hensive treatment in §11.3 of Lessler and Kalsbeek (1992). 
Since the design is self-weighting the sample mean, Y, 
is the natural estimator of the population mean. Its 
variance under the correlated response model (1) is 


V(Y) = (Ayo? + ficat + 0*)/n 


with A; = Y;n?/n, where n; is the number of respondents 
handled by the i-th interviewer andn = Yn;is the total 
sample size, and fic = Ypm,/n where m, denotes the 
number of respondents in the p-th PSU. Note that 7, is 
always larger than the simple arithmetic average of the 
n;’s and can be considerably larger if the n;’s vary widely. 

Now consider what the corresponding expected vari- 
ance, M%( Y) say, would be if the 7 observations had been 
generated independently (e.g. if we had drawn a simple 
random sample from a very large population of PSUs 
using a large pool of interviewers). It follows from (1) that 


KW = o7or/N (2) 
where 
aS = o7 ar ce + 0°. 
The inflation in the expected variance due to the com- 


bined effects of interviewer variability and intra-cluster 
correlation is given by the ratio 


Dy = V(Y)/V 


b+ Gi lee Re = lee (3) 


where p; = 07/07, and pc = o%/0%,. We shall refer to 
this ratio as the “‘design effect’’ although it differs slightly 
from the usual definition which is in terms of actual, rather 
than expected, variances. It is clear from expression (3) 
that interviewer variability can have a substantial effect 
on the variance of a sample mean if the average interviewer 
case-load, fi;, is large even if the intra-interviewer correla- 
tion, p;, is relatively small. 

Next suppose that we are interested in the difference 
between two domain means rather than a single mean. We 
might, for example, be interested in gender differences or 
in differences between two ethnic groups. In the simplest 
extension of the correlated response model (1) we might 
postulate a model of the form 


d d 
Ys = ™ : ai qa; a bs a ef) (4) 


for observations from the d-th domain. Here the means, 
u\, may be different for the two domains but the inter- 
viewer and cluster effects are assumed to be the same. 

Let p{? = n{%/n™, where n{® is the number of 
respondents from domain d contacted by the /-th inter- 
viewer and n is the total number of respondents from 
domain d. Similarly, let ¢\ = m{?/n\, where m{ is 
the number of respondents from domain d lying in the p-th 
PSU. Then, under model (4), the expected variance of 
y‘ — y) the difference between the sample means 
for the two domains, is 


V( yY@ _— y()) = 


iI 1 
(mjo7 + Mcoe + 0”) ie 5 aap (5) 


where 


1 1 
Aa (a) (b))2 
my = > (pi — Bi Pies nea (6) 


s 1 1 
Mc = »D (qi? a rl a = =) 5 (7) 
p 


If the observations had been generated independently 
the corresponding expected variance would be 


1 1 
Vi = om@| —> + <> 
n® n(®) 


so that the inflation due to interviewer variability and 
intra-cluster correlation is now 


D, = Var(¥™ — Y)/YV, 


Lip Lon Uta = lipe. (8) 


The size of the effect depends on the way the inter- 
viewers’ case-loads and the PSUs cut across the domains. 
At one extreme, when each interviewer contacts the same 
proportion of people from both domains, (i.e. when 
ps = p}$)), mis zero and the interviewer effect essen- 
tially cancels out. At the other extreme, when each inter- 
viewer sees only cases from a single domain, 77, is similar 
in size to 7; and the interviewer effect for differences is 
comparable to that for a single mean. Typically inter- 
viewers contact people from both domains and ™, is 
rather small, giving some justification to the belief that 
interviewer variability has a small impact on estimated 
differences between domains. Similar comments apply to 
the effect of clustering. 
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All this depends on the assumption that the interviewer 
and cluster effects, a; and b,, are the same for both 
domains. It is easy to imagine situations where such an 
assumption would not be at all reasonable. Some inter- 
viewers, for example, might interact very differently with 
males and females, or with members of different ethnic 
groups. A model which allows for the possibility of such 
interactions is 

Ya Opal Dio oO (9) 
where a{” and a}”) (respectively b{” and b{?)) are now 
assumed to be correlated random variables with correla- 
tion r;(rc). The naive model (4) corresponds to the special 
case in which the variances of the effects are equal and r; 
and rc are both equal to one. On the other hand, if there 
are substantial differences between the interviewer (cluster) 
effects for the two domains, 7;(rc) will be small (or even 
negative in extreme cases). In the rest of this section we 
suppose for simplicity that the variances of a{%) and a”? 
(respectively b{”) and b{”’) are equal. This may or may 
not be reasonable in practice but the simplification enables 
us to concentrate on the essential ideas. The basic form 
is similar in the more general case but the terms are some- 
what messier. Under model (9), the expected variance of 
Y@ — 0) is 


u s 1 1 
ie tee ea) (Prone. Voc. 0.) | an ha 
Ae aa 


(10) 
where 


= YS (pi = 2rpip + pO Wee = a) 


[ 


with a similar definition for ¥¢ in terms of q{” and q{”). 


The variance inflation factor under this model is 
Dy, = 1+ (Hp — er + (We — Ipc. (11) 


This is a decreasing function of r;, the correlation 
between the interviewer effects for the two domains; the 
smaller the correlation, the larger the variance inflation. 
When 7; = 1, ¥; reduces to m, and the interviewer effect 
is negligible provided all interviewers see a reasonable 
balance of people from both domains. However, if 7; is 
small (indicating a strong interaction between the inter- 
viewers and domains), #, is the same order of magnitude 
as f, and the effect of interviewer variability on the vari- 
ance of domain differences can be substantial. 

In practice, the effects will fall between the two extremes 
and their likely impact is a matter for empirical enquiry. 
In the next section, therefore, we make a start on building 
up practical knowledge about the impact using data for 
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a variety of questions drawn from a single health survey 
that is typical of the genre of research investigation for 
which domain comparisons are important (although not 
ideally designed for our purposes!). 


3. EXAMPLE 


The example is based on data drawn from a survey of 
the oral health, attitudes and practices of adult New 
Zealanders. The details of the survey are reported in full 
elsewhere (Cutress et a/. 1979). The important features of 
the study for the purposes of the current investigation are 
the sample design and the deployment of interviewers. 

The sample design was a stratified multi-stage sampling 
scheme. The country was divided into 256 Territorial 
Local Authorities (TLAs) and a geographically stratified 
sample of 68 TLAs was drawn from the 256 with selection 
probabilities proportional to size (PPS) at the first stage, 
where size was the estimated number of persons aged 
15 and over. Each sampled TLA was split into secondary 
sampling units (SSUs) comprising existing census mesh- 
blocks, aggregated where necessary in order to achieve a 
minimum size of 50. Two SSUs were then selected with 
PPS from each sampled TLA at the second stage. Finally, 
a systematic sample of 28 adults was drawn from each 
sampled SSU. This equalised the final probability of 
selection for all adults so that the sample design is (approx- 
imately) self-weighting. 

The key point of the design was the deployment of the 
interviewers. Thirteen interviewers were employed in the 
study, with at least three interviewers used within each 
SSU, and all interviewers carried out at least 10% of their 
total work-load in one region (Auckland). Ideally the 
assignment of interviewers would be part of the overall 
sample design as in Fellegi (1974) or Biemer and Stokes 
(1985). Unfortunately the study was not designed to estimate 
interviewer variance, and the assignment of respondents 
to interviewers was done in a haphazard way, rather than 
using a formal randomization procedure. 

This is fairly typical of large studies. The following 
quote from Hox (1994) gives a good summary of the situa- 
tion: ‘‘Ideally, in interviewer studies, respondents should 
be assigned to interviewers at random. In large-scale 
studies, this is seldom done because it is expensive and 
complicated to organize. This makes it difficult to use such 
studies for methodological research because, as a result, 
interviewer and respondent characteristics might be con- 
founded. Multi-level analysis, as outlined above, offers 
some remedies for this situation. If the relevant respondent 
variables are known they can be put in the regression 
model to equalize interviewers by statistical means... The 
limitation of this approach is that it relies on statistical 
control instead of experimental control. It depends on the 
assumption that all relevant covariates have been included 
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and have been correctly modeled. Without randomization, 
it is impossible to conclude that the influence of all confound- 
ing variables has been eliminated.”’ In our case, the deploy- 
ment is such that all the components of variance are formally 
identifiable, provided that we believe the model and are 
willing to accept that the assignment of interviewers is inde- 
pendent of the cluster effects. However, because of the lack 
of formal randomization there always remains the possibility 
that variations in patterns of response between interviewers 
could be a function of workload allocation rather than inter- 
viewing style. Clearly the empirical results can only be 
regarded as tentative, pointing out possibilities that will 
need to be explored further in properly designed studies. 
Even if we ignore the lack of randomization in the inter- 
viewer deployment, the study design is considerably more 
complicated than the one assumed for the development of 
the theory in the previous section, since it involves three 
stages of sampling and regional stratification of the first 
stage units. In the full analysis, we fitted a more complex 
model including fixed effects terms for the stratification, 
a hierarchical random effects model for the three stages 
of sampling, and all second-order interaction terms. How- 
ever it turned out that the TLAs used as the first stage units 
were so diffuse that the differences between strata and the 
between-TLA component of variance were negligible for 
all the variables used in the following analysis. Thus the 
between-SSU component is dominant and, for all prac- 
tical purposes, we can treat the design as if it were a two- 
stage sample with the meshblocks (aggregated where 
appropriate) as PSUs. We have ignored the other com- 
ponents in the results reported in the next section. 


4. RESULTS 


We look first at interviewer and cluster effects on a 
selection of means and proportions. We have used Model 
(9) for both types of variable. It is now well-known that 
this leads to an under-estimate of the variance components 
for binary data (see Anderson and Aitken 1985 and 
Pannekock 1988 for example), so our estimated design 
effects for proportions should be regarded as lower 
bounds. The models are fitted using PROC GLM in SAS. 
The impact of clustering has been well documented in the 
literature (Kish 1965; Kish and Frankel 1970; 1974). In 
general terms, the magnitude of the effects of clustering 
depends on the type and number of units chosen and is 
likely to vary with different kinds of social and demo- 
graphic characteristics. In the current investigation clus- 
tering effects were expected to be reasonably high because 
the census meshblocks used as sampling units are likely to 
show a fair degree of internal homogeneity. In keeping 
with this concentration of population characteristics, it 
was assumed that demographic and related items would 
show the largest values of pc. Values of p; were expected 


to be lower because of the intense interviewer training. The 
literature suggests that these effects are also likely to vary 
according to the type of questionnaire item, with attitude 
questions, questions requiring probing, fixed-alternative 
and forced-choice items, together with poorly-worded and 
ambiguous questions, being particularly susceptible to 
interviewer variability (Feather 1973, Groves 1989). 

Estimated measures of intra-interviewer and intra- 
cluster correlation coefficients for a selection of question- 
naire items falling under four separate headings (socio- 
demographic, attitudinal, reports of recent behaviour, and 
recall of distant behaviour) are outlined in the first two 
columns of Table 1. These categories were identified as 
providing natural groupings with the potential to display 
a wide range of interviewer effects. Within each grouping 
the items are listed in order of the size of their intra- 
interviewer correlations. A full description of all question- 
naire items (apart from the self-evident socio-demographic 
category) is provided in the Appendix. 

As expected, the socio-demographic variables (except 
for gender) show the highest values of intra-cluster corre- 
lation. The average pc is .07 (.08 if gender is omitted). 
The average values of oc for the other three categories of 
item are .02 and less. A few items that might be expected 
to be closely related to social background - like dental 
visiting, payment for visits, toothbrushing and certain 
attitude statements - have higher than average pc values. 
In general, though, these values fall within the range 
reported by others. (See, for example, Kish 1965, p. 581 
for a series of consumer surveys, Bebbington and Smith 
1977 and Verma et al. 1984 for the country studies in the 
World Fertility Survey.) 

The corresponding estimated p,; values are listed in the 
first column of Table 1. In general these values are very 
much smaller than those recorded for cluster effects, being 
usually less than half, and in some cases a tenth, the size 
of the pc values for the corresponding items. As expected, 
some attitude items show higher than average p; values, 
as do certain reports of behaviour that might be susceptible 
to a high ‘‘social desirability’’ bias, like toothbrushing and 
buying sweets and chocolates. Ethnic group and employ- 
ment status also record relatively high values. The pattern 
is similar to that found in previous studies, although the 
values recorded lie at the lower end of the range of typical 
values reported elsewhere (Feather 1973; Kish 1962; 
O’Muircheartaigh 1977; O’Muircheartaigh and Wiggins 
1981). A comprehensive survey is given in Chapter 8 of 
Groves (1989). This may partly reflect the intensive training 
and monitoring of the interviewers that were integral to 
the field work stage of the study. It may also be influenced 
by the rigorous post-field work ‘‘cleaning’’ (editing and 
checking) of the data that was carried out prior to analysis. 
However it may also simply be due to the attenuation 
resulting from using Model (1) for proportions that we 
noted above. 
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Table 1 


Cluster and Interviewer Effects for Means 
and Proportions 


Item Description br Ye Do % Int 
Attitudinal: 
Dentists 1 .014 .014 4.61 91 
Visiting .008 .028 3.42 74 
Natural Teeth .008 .027 3.5 76 
Health of Teeth .007 015 2.97 84 
Dentures .005 .015 2.67 80 
Dentists 2 .004 .033 Dei 57 
Health of Gums .003 .010 1.96 77 
Fluoridation .001 .016 1.66 49 
Average .006 .020 2.95 73 
Socio-demographic: 
Employment Status .010 .055 4.20 65 
Race .009 72 6.87 33 
Age .004 .042 5.98 52 
Household Income .002 .092 B29) 15 
Marital Status .000 .058 2.34 0 
Sex of Respondent .000 .005 La 0 
Average .004 .071 3.47 28 
Recent Behaviour: 
Brushed Teeth .019 .025 6.16 8 
Sweets/Chocolates O11 .003 36S 98 
Fluoride Toothpaste .008 .000 3.04 100 
Toothpick .006 .006 2.66 92 
Rinse Mouth .004 .024 2.43 62 
Dental Floss .001 .018 1.60 43 
Disclosing Tablet .000 .027 1.49 0 
Mouthwash .000 .018 1.42 0 
Average 006 012 2.82 60 
Distant Behaviour: 
Age First Paid .004 .029 2.34 5y/ 
Visited Dentist .004 .029 Dei 57 
Cost Last Year .002 .000 1.19 100 
Year Last Visit .000 .014 ilstlSs 0 
Average .003 -018 1.80 54 


Perhaps more significant than the pattern and values 
of p; is the impact of interviewer variability on the overall 
design effect, incorporating both interviewer and clustering 
effects. This is shown in the third column of Table 1 
(Do), with the final column (% Int) representing the pro- 
portionate contribution of interviewer variability to the 
overall value of Dp. Design effects are substantial, being 
above two in all but a minority of cases. This is due to the 
clustering and to the impact of the large interviewer work- 
loads characteristic of the study since, from equation (3), 
the variance is increased by a factor of | + (A; — 1)p,, 
where fi; is a weighted average of the interviewer work- 
loads. There is a distinct pattern in the contribution to the 
design effect produced by interviewer variability. For socio- 
demographic variables it averages just under one half of 
the contribution from clustering, while for attitudinal 
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items the interviewer contribution to the design effect rises 
to three times that from clustering. The other two cate- 
gories of items range in between these two extremes. 

What the results outlined in Table 1 confirm is the 
impact that interviewer workload has on the variance of 
sample estimates, because of the multiplier effect. In 
essence, an interviewer component with a very small intra- 
class correlation can be translated into a major effect if 
the interviewer workload is high. In the study under 
review, the logistics of deployment and the requirements 
of on-going quality control seemed to argue for small 
interview teams, a practice that appears to be typical of 
much field work in the health area (for example, Choi and 
Comstock 1975). This meant that interview workloads 
averaged over 250. The cost of this strategy is immediately 
apparent from the results in Table 1; very small differences 
between interviewers are translated into major reductions 
in the precision of sample estimates. 

Now we turn to the main object of our analysis, viz. the 
impact of interviewer variability on contrasts between 
domain means or proportions. In the current analysis, this 
was assessed for two sets of comparisons, the first set by 
gender (male/female) and the second set by ethnic group 
(European/non-European). As we have seen in the discussion 
following equation (11), the contribution, 1 + (v; — 1)p,, 
to Do from interviewer differences depends on the extent 
to which the interviewer effect is constant across the two 
domains and on the way the domains cut across individual 
case-loads. Assuming that the domains cut evenly across 
interviewer case-loads, then v, is zero if the interviewer 
effect is identical in the two domains, in which case the 
common interviewer effect cancels out completely in the 
comparison. On the other hand, if the effects in the two 
domains are weakly correlated then the value of v,; tends 
to be much higher and in extreme cases may equal the 
average case-load. In the current study values of v, fell 
between 0 and 50 for both gender and ethnic group. Thus 
the effect of domain-specific interviewer effects on the 
design effect can be quite substantial. Similar comments 
apply to the impact of clustering on the comparison; if the 
effect is the same on both domains then it largely cancels 
out and the net impact is small, but the impact can be 
substantial if the clustering effect is domain specific. 

Table 2 shows values of 9; and pc for comparisons by 
gender and by ethnic group, together with the overall design 
effect D, and the proportion of this effect due to the 
impact of interviewer variability. Note that the item on the 
use of disclosing tablets has been omitted from Table 2. 
This is because so few respondents either used or knew 
what this item was that the effective sample size in this case 
is tiny, thus rendering the results almost meaningless. 

The impact of both interviewer and clustering on compar- 
isons by gender is small with design effects little above unity, 
in spite of the fact that the estimated values of p; and pc 
are slightly increased when adjusted for this variable. 
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Table 2 


Interviewer and Cluster Effects for 
Domain Differences 


By Sex By Race 
Item Description 


Attitudinal: 
Dentists 2 .004 .043 1.05 0 01010271228 46 
Visiting -009 .028 1.08 0 (O72 5213305319 78 
Natural Teeth .010 .032 1.06 0 010 .037 1.26 42 
Fluoridation .001 .019 1.12 42 O21 20312218 88 
Dentures .007 .018 1.04 0 AOL ORY A 33 
Health of Teeth O12" 2022" 152 85 010 .045 1.40 20 
Dentists | 001 .018 1.05 0) 10159020" 1653 74 
Health of Gums 006 .022 1.07 0 .003 .104 1.46 9 
Average 006 .025 1.12 16 019 .054 1.93 49 


Socio-demographic: 


Race SOKO ate i bal 0 - - - 
Household Income .004 .095 1.37 24 004 .099 1.95 40 
Marital Status .000 .059 1.17 0 011 .060 1.69 38 
Employment Status .014 .067 1.42 71 O22 1G 2.09 25 
Age 007 .052 1.06 0 .006 .093 1.87 24 
Sex of Respondent - - - 006 .011 1.09 44 
Average 007 .091 1.23 19 .010 .076 1.74 34 
Recent Behaviour: 
Brushed Teeth 025 .060 1.62 65 019.019 1.68 88 
Rinse Mouth .007. .029 1.28 64 .004 .023 1.20 45 
Mouthwash .000 .0S7 1.20 0 .027  .105 2.69 7) 
Dental Floss .003 .021 1.06 0 2015) .036) 1537 32 
Toothpick .006 .010 1.03 0 006 .046 1.48 63 
Sweets/Chocolates .012 .009 1.02 0 ADS} ORY SLASH 48 
Fluoride Toothpaste .010 .007 1.11 100 007 .000 1.02 100 
Average -009 .028 1.19 33 013 .036 1.36 64 
Distant Behaviour: 
Age First Paid 10037 2033) 1210 0 .029 .141 2.92 71 
Visited Dentist 005 .035 1.04 0 L020 OLS) 21026 50 
Year Last Visit .004 .012 1.20 75 -016 .003 1.83 12 
Cost Last Year .007 .021 1.01 0 O76 DL 72.09 42 


Average 005 .025 1.09 19 -035 .070 2.03 44 


A significant gender-specific effect was apparent for only 
three items, health of teeth and tooth-brushing - for which 
there may be a unique social acceptability bias - and 
employment status - which holds quite different connota- 
tions for men and women. Note that the interviewer effect 
is the dominant one in all three of these comparisons. 

The impact on comparisions by ethnic group is much 
higher, with design effects averaging about 1.7. This 
suggests that there are significant, non-cancelling inter- 
viewer and clustering effects associated with the ethnic 
identity of respondents. There are large ethnic-specific inter- 
viewer effects for two hypothetical attitudinal questions 
(visiting and fluoridation), for one item of recent behaviour, 
and for age of first payment for dental services. The result 
is plausible; all the interviewers were European and may 
have varied systematically in their interactions with re- 
spondents of different ethnic backgrounds. Again clus- 
tering effects are most marked for the socio-demographic 
variables. Not only are the design effects on average higher 
than those recorded for the gender comparisons, but the 
interviewer component is in general two or three times 
higher for the ethnic group contrasts. 


py Po D, “%lnt p; poe D, Mint 


A referee rightly points out that because of the way the 
interviewers are deployed (they worked primarily in teams 
assigned to different parts of New Zealand), there is a real 
possibility that the interviewer effects might be inflated 
because of confounding with area effects. The fact that 
differences between the TLAs were so small gives us some 
reason to believe that this inflation will be small, but the 
possibility can never be discounted with this design. 


5. DISCUSSION 


This paper has applied empirical data from a not un- 
typical health survey to assess the impact of interviewer 
variability under the assumptions of both simple and 
extended versions of the correlated response model for the 
error variance of a multi-stage sample design. 

In the first case the simple model analyses the relative 
impact of cluster and interviewer effects on the estimation 
of means and proportions. The results of this analysis 
confirm a number of findings that are well established in 
the literature: the intra-class correlations for interviewers 
are generally lower than those for clusters; the intra-class 
correlations for clusters vary in the expected direction by 
question type; the overall design effects for these question 
types vary between 2 and 3.5; a substantial component of 
this inflation is contributed by interviewer variability and 
can probably be attributed to the multiplier effect of large 
interviewer caseloads; finally, the impact of this inter- 
viewer component is shown to vary in the expected direction 
by question type. 

In the second case the extended model addresses the 
analysis of cluster and interviewer effects for the estimation 
of domain contrasts between means and proportions for 
two sets of comparisons defined by gender and ethnic 
group. The effect on contrasts between domain means was 
smaller but it was still significant for a number of items, 
particularly for the ethnic comparisons, suggesting that the 
interviewer effect was different for the two domains. The 
size of the effect for these items was certainly large enough 
to suggest that we should be concerned about it in designing 
similar studies. In general, the impact of interviewer 
effects was two or three times as great for the ethnic 
contrasts as it was for the gender comparisons. 

The basic deficiencies in the design mean that these 
results must be regarded as suggestive rather than defini- 
tive. They do indicate, however, that there is considerable 
potential for damage in the use of a small group of inter- 
viewers even when interest is centered on domain differ- 
ences rather than simple means or proportions. This is 
certainly counter to standard folklore in some fields such 
as health surveys, and suggests that considerable further 
empirical work is justified. 

On the assumptions of the simple correlated response 
model a reduction in the impact of interviewer variance 


Survey Methodology, December 1995 


can be achieved by raising the number of interviewers and 
thus reducing individual interviewer workloads. Of course, 
this brings with it a potential reduction in the quality of 
interviewing if training and monitoring procedures have 
to be tempered. In this instance close attention to question 
wording and interviewer instruction is clearly crucial. In 
the case of the extended version of the correlated response 
model, however, such a strategy is unlikely to be a suffi- 
cient one on its own. If comparisions between groups are 
a major objective of the study, then it is important also 
to ensure that the interviewers treat the two groups in as 
similar a way as possible. It is also important to design the 
study so that each interviewer contacts respondents drawn 
from both groups. This is likely to be a critical considera- 
tion in investigations such as case-control studies in which 
health outcomes are related to contrasting exposures and 
in which the control of potential confounder variables may 
have a significant influence on the magnitude of measures 
such as the odds ratio. 
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APPENDIX 
Questionnaire Items 


Attitudinal 

Dentists 1: ‘‘Dentists are more interested in their 
patients than making money.”’ 

Dentists 2: “Dentists recommend a lot more things 
to be done than really need to be done.”’ 

Dentures: “‘Dentures are just as good (or better) 
than your own teeth.”’ 

Fluoridation: ‘“What is your opinion on fluoridating 
public water supplies?”’ 

Visiting: “Do you think a person should go to the 


dentist only when they have dental prob- 
lems or should they go sometimes also 
when they have no obvious problems?”’ 


Health of Teeth: ‘‘If you went to the dentist tomorrow, 
do you think he would find anything 
wrong with your teeth?’’ 


Health of Gums: ‘‘If you went to the dentist tomorrow 
do you think he would find anything 
wrong with your gums?’’ 
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Recent Behaviour 


“Yesterday did you - use a dislosing tablet/mouthwash/ 
dental floss/toothpick? 
— rinse after eating? 
— brush your teeth?’’ 


‘Did you buy sweets or chocolates any time last week?”’ 


Distant Behaviour 


Age First Paid: ““About how old were you when you 
first went to a dentist for routine 
treatment for which you or your 


family had to pay?”’ 


Visited Dentist: “Did you visit a dentist in the last 


12 months?’’ 


Year Last Visit: “‘In what year did you last visit a 


dentist?’’ 


Cost Last Year: ““About how much did you pay 
for dental treatment in the last 


12 months?’’ 
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On Searls’ Winsorized Mean for Skewed Populations 


LOUIS-PAUL RIVEST and DANIEL HURTUBISE! 


ABSTRACT 


This paper considers the winsorized mean as an estimator of the mean of a positive skewed population. A winsorized 
mean is obtained by replacing all the observations larger than some cut-off value R by R before averaging. The 
optimal cut-off value, as defined by Searls (1966), minimizes the mean square error of the winsorized estimator. 
Techniques are proposed for the evaluation of this optimal cut-off in several sampling designs including simple 
random sampling, stratified sampling and sampling with probability proportional to size. For most skewed distribu- 
tions, the optimal winsorization strategy is shown, on average, to modify the value of about one data point in the 
sample. Closed form approximations to the efficiency of Searls’ winsorized mean are derived using the theory of 
extreme order statistics. Various estimators reducing the impact of large data values are compared in a Monte Carlo 


experiment. 


KEY WORDS: Outliers; Max domain of attraction; Mean square error; Simple random sampling; Stratified 


sampling. 


1. INTRODUCTION 


Samples drawn from positively skewed populations 
often contain outliers with values that are much larger than 
most sampled values. One usually tries to accomodate 
these large values when designing the survey (Glasser 1962; 
Hidiroglou 1987). However, given the multipurpose nature 
of most surveys, statisticians are often faced with outliers 
at the estimation stage. These data points make classical 
survey estimators, such as the sample mean, unstable. It 
is therefore of interest to study alternative estimators that 
lower the impact of large data values. Winsorization 
(Searls 1966) consists in replacing the data values larger 
than a cut-off value R by R before averaging. Searls 
suggested to select the value of R which minimizes the 
mean square error of the winsorized mean. One can also 
take R equal to the second largest data value in the sample 
(Rivest 1994). Searls’ estimator was best among all the 
methods to adjust large data values studied by Ernst 
(1980). Hicks and Fetter (1993) implement Searls’ winsor- 
ization strategy in an agriculture survey. Other strategies 
have been proposed for dealing with large observations in 
survey sampling. Chambers and Kokic (1993) review esti- 
mators derived from the theory of ‘‘Robust Statistics’’ 
(Huber 1981). Fuller (1991, 1993) proposes a preliminary 
test to detect the presence of extreme values in the sample; 
the impact of these values is lowered only in samples for 
which this test is significant. Lee (1994) provides a good 
review of this expanding literature. 

The key to the implementation of Searls’ winsorization 
method is the selection of the cut-off R. A simple algorithm 
for calculating the optimal cut-off for a known population 


in simple random sampling and in pps sample is proposed 
in Section 2. Repeated calculations of the optimal cut-off 
for several populations and several sample sizes reveal 
that, in most cases, the optimal scheme winsorizes one data 
point on average, regardless of the sample size. Section 3 
extends the result of Section 2 to stratified sampling. A 
simple algorithm for the calculation of cut-off values in 
each stratum is proposed. The rule of winsorizing an 
average of one data point per sample regardless of sample 
size is shown to hold also in stratified samples. The effi- 
ciencies, with respect to the sample mean, of various 
winsorized estimators are calculated in Sections 4 and 5S. 
Section 4 derives analytic large sample approximations to 
the efficiency of Searls’ estimator using the theory of 
extreme order statistics while Section 5 compares, in a 
Monte Carlo study, estimators for reducing the impact of 
large data values. 


2. SAMPLING PROPERTIES OF THE 
WINSORIZED MEAN 


This section studies winsorized means for data drawn 
from either a continuous or a discrete distribution. Several 
families of continuous distributions are available to model 
positive skewed data. One has the Weibull family, F(x) = 
1 — exp(—(x/6)'/*) for x > 0, the log-normal family, 
F(x) = ®(log(x/B8)/v) for x > 0, and the Pareto family, 
F(X) te Xe)? forex: > Oe wheres Mista 
positive scale parameter and a, v, and y are positive shape 
parameters. Discrete skewed distributions arise in survey 
samping. Let {y,, ..., ¥y} represent the values of the 
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variable of interest for the N units of a population to be 
sampled. If a simple random sample with replacement is 
drawn, then one can take F(x) = ¥J(y; S x)/Nas the 
underlying distribution where /( - ) represents the indicator 
function. In pps sampling, i.e., sampling with replacement 
and with probabilities given by {p;, i = 1, ..., N}, one 
would take F(x) = Y¥p;l(y;/(Np;) <= x). The standard 
estimator of y under pps sampling, 


can then be regarded as the mean of a random sample of 
size n drawn from distribution F. Fuller (1991) provides 
examples of survey data having skewed distributions. 
Let X;, X>, ..., X, denote a sample drawn from F(x). 
In pps sampling, one would have X; = y;/(Np;) where p; 
and y; are the selection probability and the value of the 
y-variable for the i-th unit selected in the sample. The 
population mean p is to be estimated by a winsorized mean, 


= (gees weitia Res 
Xp = min(X;,R) =x — = max (X; — R,0), 
R we (X;,R) ye ( ) 
= (ail) 


where X is the mean of the X;’s. The expectation of Xp 
is equal to 


fore) wo CX 
EUxey =o =) (x — R)dF(x) = p -| dydF(x). 
R RJR 


Changing the order of integration in the above integral 
proves that E( Xr) = » + B(Xp) where 


oo 


[1 — F(x) ]dx (2.2) 
R 


B(Xp) = >| 


is the bias of the winsorized mean. 
By (2.1), an expression for the variance of Xp is 
nVar(Xp) = 0? — 2cov[X,,max(X, — R,0)] 
+ Var[max(X, — R,0)] 
where X; is the first random variable in the sample and 


o° is the variance of F(x). Manipulations similar to those 
yielding (2.2) show that 


lo) 


E[max(X; — R,0)7] = 2 (x-— Ril aid 
R 


and 


E[max(X; — R,0)X,] = 


2|" ex SR} T= F(x) lee RB Ce). 


R 

Thus 

VanGXe) 
fo? or 2\" ex =—itpt i tee MOG) ais Bite}, 
n R 


and 


oO 


MSE(Xp) = 


IS) 
Pa a) 


Px (x =p) [1 =akGeitdx 
n—-1 


+ B7 (Xp 2) 


Searls (1966) showed that the mean square error of Xp 
has a unique minimum which can be obtained by equating 
the derivative, with respect to R, of MSE( Xp) to 0. This 
yields the following equation for the optimal winsorization 
constant R(F,n), 


Aaa =a — F(x)]dx = 0. (2.4) 


This is equivalent to equation (14) in Searls (1966). In the 
remainder of this work, Xp denotes the optimal winsorized 
mean obtained with the winsorization constant R(F,n) 
which solves (2.4). Observe that the optimal cut-off point 
R(F,n) is location and scale equivariant, /.e.,if G(x) = 
F[ (x — b)/a], then R(G,n) = aR(F,n) + b. 

A general algorithm for solving (2.4) is easily con- 
structed. First observe that as a function of R, the left hand 
side of equation (2.4) is increasing and concave in R since 
its derivative, 1/(n — 1) + 1 — F(R), is positive and 
decreasing. Therefore, the Newton-Raphson algorithm 
(Thisted 1988, 164-167) given by 


(Rw —(n— 0 [1 — F(x) ]dx 
Rj 

Ri =R— 

fais od 1+ (n—1)[1 — F(R)] 


(23) 


with Ro = 2 as starting value converges smoothly to the 
solution of (2.4). For discrete distributions the computa- 
tions are easily implemented by noting that 


jo — F(x)|]dx = E[max(X — R,0)]. 
R 
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Exact calculations of the optimal cut-off points R (F, 1) 
were carried out for the Weibull, the log-normal, and the 
Pareto families for samples of size s ranging between 5 and 
200. Three distributions, corresponding to coefficients of 
variation (CV) of 1, 2, and 4, were considered in each 
family except for the Pareto family where only coefficients 
of variation of 2 and 4 were considered. The CV measures 
the skewness of a distribution, with large CVs corresponding 
to heavy skewness. The corresponding parameter values 
are given in Table 1. 


Table 1 


Parameter values of the distributions for which optimal 
cut-off values R(F,n) were evaluated 


CV Weibull(a) Log-normal(v) Pareto (y) 
1 1 0.83 - 
2 1.84 L247) 2.67 
4 2.87 1.68 2eil8} 


For each distribution and each sample size, the optimal 
cut-off point was calculated using algorithm (2.5). Figure 1 
presents the expected number of winsorized observations, 
m(F,n) = n{1 — F[R(F,n)]} asa function of n while 


Pareto 
Weibull 
Lognormal 


Expected number of winsorized observations 


50 


Figure 1. 
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the corresponding efficiencies are reported in Figure 2. The 
efficiency of Xp is defined as Var (X)/MSE(X,). 

In Figure 1 the expected number of winsorized data values 
under the optimal scheme is, for most skewed distributions, 
close to | even for large sample sizes. Approximating this 
number by a Poisson distribution with parameter m(F,n) 
shows there is a non-negligible probability that, under the 
optimal winsorization scheme, none of the data points is 
winsorized. This probability increases with the skewness 
of the distribution since m(F,n) decreases with the CV. 
Thus, in samples from a highly skewed distribution, it is 
not always appropriate to winsorize the largest values. 
Such values should be winsorized only when they are large. 
As expected, in Figure 2, the largest gains in efficiency are 
obtained when the skewness is heavy. Therefore moni- 
toring the two or three largest data values in a sample and 
curtailing their impact when these values are large is the 
key to a good winsorization strategy. 

Figure | shows that the expected number of winsorized 
data values m(F,n) decreases with the skewness of the dis- 
tribution. This observation can be turned into a rigorous 
mathematical result. To this end, random variable Y is said 
to be more skewed than random variable X if Y has the 
same distribution as y (X) where (x) is a convex function 
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Expected number of winsorized observations for simple and stratified random sampling. 
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Figure 2. Efficiency of Searls winsorized mean. 


of x. Under this definition X° is, as should be expected, 
more skewed than X. This notion of skewness corresponds 
to the convex partial ordering of van Zwet (Barlow and 
Proschan 1981). With this definition of skewness, one has 
the following proposition which is proved in the Appendix 
together with Propositions 2 and 3. 


Proposition 1 If Y is more skewed than_X then m(Fy,n) > 
m(Fy,n) where Fy and Fy are the distributions of X and 
Y respectively. 

The results of this section also apply to simple random 
sampling without replacement. For this design the mean 
square error of Xp is given by formula (2.3) with n 
replaced by n/(1 — f) where fis the sampling fraction. 
Algorithm (2.5), with n divided by (1 — /), can be used 
for calculating optimal cut-off values for without replace- 
ment simple random sampling. 


3. WINSORIZATION IN STRATIFIED 
SAMPLING 


There are many ways to generalize Searls’ winsorization 
strategy to stratified sampling. In this section each stratum 
has its own cut-off value. Let R, be the cut-off value in 


stratum fh. The optimal values of R;, Ro, ..., Ry, where 
L is the number of strata, are the ones that minimize the 
mean square error of Xp = ¥ W,Xp,, where Xpp = 
¥ min(X,;,R,)/N,, W, = N,/N and N,, is the size of 
stratum hand N = YN,. An algorithm for determining 
these optimal cut-off values is proposed in this section. 

Let F,(x), forh = 1, ..., L be the distribution of X 
in stratum h, and p, and of be the mean and the variance 
of F;,. The derivation of the mean square error of Xp, 
under with replacement stratified random sampling, 
follows that presented in Section 2, it gives 


L 2 

z W; 
MSE(Xz) = )) ee 
pre 


G —2 | (x — py) [1 — Fy(x) dx - (Xe) 


Rp 


L 2 
+ (x WB (Xen) ) (3.1) 


h=1 


where B(Xp,,) is the bias of Xp, as an estimator of p, 
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B(Xpr,) = — | [1 — F(x) | ax. 
Rh 


Taking the partial derivatives with respect to R,,h = 1, 
...,L yields the following equations for the optimal values: 


W,, z ti z 
ae [Rn — Bn — B(XpRn)) = - sD W,B(Xprn), (3.2) 
h h=1 


TOU emai, ear. Ls. 

There is no simple way to solve (3.2). An approximate 
solution can be obtained by noting that B(Xp,,) /n,, is, for 
all values of 4, usually small as compared to the other 
terms. Dropping these terms leads to 


W,, a Zs 
ra (Rp -— ba) = - y W,,B(Xrn), (3.3) 
A h=1 


for h = 1, ..., L. The solutions to (3.3) overestimate 
slightly the optimal values satisfying (3.2) since at these 
solutions the partial derivatives of (3.1) are all positive and 
since these partial derivatives are increasing functions of 
R,, forh = 1, ..., L. Thus by solving (3.3) to estimate 
the cut-off values one does not run the risk of winsorizing 
too many data values. Equations (3.3) imply that R, = 
Lyn + n,R/(nW,,) where R is some positive constant. A 
simple equation for R is obtained by changing variable 
y = nW,(x — p,)/n, in the integrals for B(Xp,), 
i=. Lawicien? =) 71;,. 1 nis pives 


m e R 
» W,B(XrR,) = = = 
h=1 


[1 — F(y)]dy = — B(XR), 3.4) 
R 


where F(y) = Yn,Fylun + nny/(nW,,)]/n. Equation 
(3.4) is easily solved using algorithm (2.5) proposed in 
Section 2 for the single sample case. Therefore simple 
approximations for Searls’ optimal cut-off values in 
stratified sampling are easily calculated. 

Since the distribution F defined above has a zero expec- 
tation, the mean square error of the stratified winsorized 
mean obtained by solving (3.3) is equal to: 


7 1 
MSE(X,) = — 
n 


Ge 2 5 ALU oy UNE D ae Bt&,)”) + B(Xp)* 
R 


L 2 p2/ ¥ 
iy (eesieeiseeaye BSED) si 
n n 


h=1 h 


Wi 


where of is the variance of F. The last term of (3.5) 
is easily shown to be negative or null; it is null when 
B(Xp) = nW,B(Xp,,)/ny, for h = 1, ..., L. The variance 
of the stratified mean, ¥ = Y W,,X,,, is equal to of/n. 
Thus a conservative approximation to the efficiency of 
Xp with respect to X in stratified sampling is equal to the 
corresponding efficiency for a random sample of size n 
drawn from F. Note also that n[1 — F(R)] represents 
the expectation of the total number of winsorized data 
points in the L strata. 

The optimal winsorization scheme obtained by solving 
(3.3) has a simple form for many allocation rules. Under 
proportional allocation, i.e.,n, = nW,forh = 1,..., 
L, one gets R, = pu, + R. Under Neyman optimal alloca- 
tion, with n, = nW,,o,/(¥ W,,o,) where o, is stratum h’s 
standard deviation, one gets R, = up, + o,R/(¥ Wo,). 
If in addition, the distributions of X within the strata are 
equal up to a change in location and scale, i.e., F;, = Fo 
[ (x — p,)/o,] for some distribution Fo, then F(x) = 
Fo[x/(¥ W,o;,) |. In this case the characteristics of optimal 
winsorized means in stratified sampling and in simple 
random sampling are the same. Thus Figure 1 presents 
the expected total number of winsorized data points in 
the L strata as a function of the total sample size n, 
under Neyman allocation, when Fo is one of the distri- 
butions of Table 1. Figure 2 gives the corresponding 
efficiencies. 

The results of this section are easily generalized to 
without replacement stratified sampling by replacing n, 
by n,/(1 — f,) throughout the calculations. The deriva- 
tion of optimal cut-off values for stratified pps sampling 
is easily carried out by taking F,(x) = Y¥ ppl (yi / 
(Np Pri) < X) where p,; denotes the selection probability 
for unit the /-th unit of stratum h. 


4. LARGE SAMPLE APPROXIMATIONS TO 
THE EFFICIENCY OF THE 
WINSORIZED MEAN 


For most distributions, equation (2.3) defining the 
optimal cut-off does not have an explicit solution. This 
section derives closed form approximations to this solution 
using the theory of extreme order statistics. This will permit 
the derivation of explicit approximations to the efficiency 
of the optimal winsorized mean. Searls’ optimal winsor- 
ization strategy will then be compared to a simple non 
parametric winsorization scheme where the largest order 
statistic is replaced by the second largest (Rivest 1994). 

The form of the approximation to R (F,n) depends on 
the limiting distribution, as the sample size n goes to 
infinity, of the largest order statistic suitably normalized. 
For distributions whose support is the positive axis, there 
are only two possible limiting distributions which are given 
by Galambos (1987, p. 53-54) 
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For many distributions used for the statistical analysis of 
positive random variables, for example the Weibull and 
the log-normal families, the sample maximum suitably 
normalized converges to H39(x). Distributions whose 
sample maxima converge to H, ,(x) for some a > 0 
have heavy tails. For such distributions 1 — F(x) goes to 
0 at arate of O(x “). The Pareto and the F distributions 
are in this class. 

Distributions whose sample maxima converge to 
H3,9(x) are considered first. The following characterization 
is due to von Mises (1964): the sample maximum of a twice 
differentiable distribution F(x) converges to H3 (x) if, 
as X goes to a, 


ge(x) 
- =0 (4.1 
m 92 (x) 


where f(x) is the density of F, g(x) = f(x)/[1 — F(x) ] 
is the failure rate of F, and g’ is the derivative of g. An 
approximation to winsorization constant R (F,n) for this 
class of distributions is presented next. 


Proposition 2 If F(x) is such that (4.1) holds and if, 
for large values of x, it satisfies: 


i) xg(x) increases; 
ii) xg’ (x)/g(x) is less than some positive constant c; 


then the optimal winsorization constant R (F,n) satisfies 


IRGRD) = 


> 


re Bini peli eeidles eee 


n 


ands mF, ak=re (F114 n)) Be hele adn) 
[1 + o(1)]. Furthermore, the mean squared error of 
Searls’ winsorized mean is approximately equal to 


2 2 

a R(F, 

MSH Cried ese ul 
n n 


In the Weibull family, F7'(1 — t) = [—log(t)]%, 
g(x) =x!/*—!/q. The hypotheses of Proposition 2 are met 
and m(F,,,n), the expected number of winsorized obser- 
vations in a large Weibull sample, is log(n) [1 + 0(1)]/a 
which goes to o as n increases. Figure | suggests that the 
convergence is very slow, especially for large coefficients 
of variation. 


Now consider distributions whose sample maxima con- 
verge to H, ,(x). This class of distributions has been 
characterized by Gnedenko (1962): the sample maximum 
of F converges to H, ,(x) if one can write 


to AX) Se i (4.2) 


where as x goes to o, L(x)/L(kx) converges to 1, for 
any constant k. Note that for F to have a finite second 
moment, one needs a > 2 in (4.2). The Pareto distribu- 
tion satisfies (4.2) witha = y. 


Proposition 3. If F satisfies (4.2) with parameter a 
where a > 2, then as n goes to infinity, R(F,n) = 
F-![(1 —(@—1)/n] [1 + o(1)], ie., m(F,n) ~a—-1. 
Furthermore, 


2 o aR(F,n)* 
MSE(Xp) = — - 5 
n n°“(a — 2) 


For distributions satisfying (4.2) a finite number of data 
points are on average winsorized as the sample size goes 
to oo. To some extent, this can be seen in Figure 1 where 
the curves of m(F,,n) for the Pareto distribution have 
M(F,33,n) = 1.33, and m(F67,n) = 1.67 as asymptotes. 

Propositions 2 and 3 shed some light on the estimation 
of the optimal cut-off value. When F is unknown, a 
possible estimator for R(F,n) is the value that minimizes 
an estimator of the mean square error of Xp. This leads to 


R-—X 
n— 1 


tpl by max(X; — R,0) (4.3) 
n 


i=1 


as an estimating function for R. This procedure is ques- 
tionable when the underlying distribution is highly skewed, 
i.e., when F satisfies the assumption of Proposition 3. On 
average, there will only be a — | non-null terms in the 
right hand side of equation (3). Thus R will, on average 
be determined by the a — 1 largest data values and the 
sample maximum will have the largest influence on R. This 
will make R highly unstable and, considering the findings 
of Figure 1, the second largest sample order statistic should 
be a better estimator of R(F,n) than the solution of (3.3). 
This is exemplified in the Monte Carlo simulations of 
Section 5. 

Table 2 compares approximations to the bias and to the 
mean square error of Searls’ winsorized mean Xp to those 
of the once winsorized mean X, obtained by taking the 
cut-off value R equal to the second largest observation. 
Rivest (1994) shows this choice of cut-off value yields the 
optimal non-parametric winsorized mean. He also derives 
the large sample approximations for the bias and the mean 
square error of X, appearing in Table 2. The corresponding 
expressions for Xp are taken in Propositions 2 and 3. 
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Table 2 


Approximations to the bias and to the mean square error of the once winsorized mean XxX 
and of Searls’ optimal winsorized mean, Xp, for the Weibull and for the 
Pareto distribution (I'(-) stands for the gamma function) 


WEIBULL 
D, 2a 
re log n 
Ye A MSE tie G UO8 x 
n n 
log n)* 
bias ee 
n 
2 Neg 
ws 2 — 1)(log n 
x, MSE aes a(a At gn) 
n n 
log n)®7! 
Fine pales 
n 


In Table 2 the mean square error of Xp is much smaller 
than that of X,. Indeed, for the Weibull distribution the 
large sample efficiency of Xp with respect to_X; is equal 
to that of Xp with respect to X. Thus non-parametric 
winsorization reduces the mean square error of estimators 
of the mean of a skewed population however further 
reductions in mean square error can be obtained if infor- 
mation concerning the underlying distribution is available. 
This is illustrated in the Monte Carlo comparisons presented 
in the next section. 

The results of this section apply to stratified sampling. 
For this design, the large sample solution to equation 
(3.4) is determined by the stratum with the most skewed 
distribution. If F, is the most skewed distribution then 
nW,R (F;,n;)/n, is an approximate solution to (3.4) where 
an approximation to R(F;,7,) is found in Proposition 2 
or in Proposition 3 depending on the tail of F,. In this 
case only data points in stratum | are winsorized in large 
stratified samples. Searls’ winsorized mean is then equal 
to W, times the optimal winsorized mean for stratum one 
plus a weighted sum of the sample means in the other strata. 


5. MONTE CARLO COMPARISONS 
OF ESTIMATORS OF THE 
MEAN OF A SKEWED DISTRIBUTION 


This section presents Monte Carlo comparisons of the 
mean square error and of the biases of five estimators of 
the mean of population CHICKEN of Fuller (1991). This 
population has 2000 units; its coefficient of variation is 


PARETO 


7g uh 


n (y — 2)(y — 197%? 77 


1 


(y x: 1) alin wy 


ao DE (ie 2/4) 
no ¥(y — In? 
(le 174) 

Siew 


4.46. Further numerical comparisons of the five estimators 
considered in this section for other distributions, either 
finite or infinite, are presented in Rivest (1993a and b). 


The five estimators under consideration are: 


e Searls’ winsorized estimator, Xp, calculated as if the 
the underlying distribution was known; 


e A winsorized estimator where the cut-off value is set 
equal to the second largest data value of an auxiliary 
sample of size 2n; this is an instance where limited 
auxiliary information concerning the underlying distri- 
bution F is available (in the Monte Carlo simulations 
each simulated sample had its own auxiliary sample); 


¢ The once winsorized mean, X\, introduced in Section 4; 


e A winsorized estimator where R is estimated from the 
sample by solving equation (4.3); 


e Fuller preliminary test estimator with j/ = 3 (7.e., the 
numerator of the preliminary test involves the three 
largest observations), 7 (the total number of data points 
involved in the preliminary test) equal to [4n” — 10] 
and K3, the cut-off value equal to 3.5. A detailed 
description of this estimator appears in Fuller (1991) and 
in Rivest (1993a and b). This estimator curtails the 
largest data values only when a test statistic for detecting 
extreme data values is significant. 


The biases and the efficiencies of Xp were calculated 
exactly. For the other estimators, the biases and the effi- 
ciencies presented in Figures 1 and 2 were obtained in 
Monte Carlo simulations based on 100,000 repetitions. 


114 Rivest and Hurtubise: On Searls’ Winsorized Mean for Skewed Populations 


Relative bias 


50 100 


R estimated from auxiliary data 
Once winsorized mean 
Preliminary test estimator 

R estimated from the sample 
Searls’ estimator 


150 200 


Sample size 


Figure 3. Relative bias of five estimators for the mean of CHICKEN. 


Figure 3 indicates that the biases of winsorized esti- 
mators are important, even in large samples. Several 
interesting conclusions can be drawn from Figure 4. First, 
as expected from Table 2 Searls’ estimator is much more 
efficient than the once winsorized mean. Estimating the 
optimal cut-off value using limited auxiliary information 
is highly efficient. This holds true as long as the study 
variable can be modeled by a superpopulation distribution 
having a finite variance, see Rivest (1993a) for further 
discussions. In a sampling context, the auxiliary samples 
could be data from previous surveys standardized to 
account for possible changes over time in the distribution 
of the variable under study. 

Among the three estimators of Figure 4 that do not rely 
on auxiliary information, Fuller estimator is the best. This 
is in agreement with the simulation results of Fuller (1991). 
Estimating the cut-off value by minimizing an estimate of 
the mean square error does poorly especially in small 
samples. Thus, as shown in Section 4, the resulting esti- 
mator is highly sensitive to the wild data values that 
sometimes appear in small samples. This estimator is not 
recommended. 


6. CONCLUSIONS 


Many strategies can be used to accomodate the large 
values that sometimes arise in surveys. If auxiliary infor- 
mation, such as census data, is available then one can use 
Searls’ estimator in either simple random sampling, 
stratified sampling, or pps sampling. Since the cut-off 
values are fixed constant mean square error estimators can 
be derived from formulae (2.3) and (3.1). 

When extra information is not available, the once 
winsorized mean and Fuller preliminary test estimator can 
be used. Research is now under way to generalize these 
estimators to stratified designs. An estimator for the mean 
square error of the once winsorized mean is proposed in 
Rivest (1994), 


~ 1 1 & 
v(X};) = — so mae, pa) ZX) 
(X,; SS 3X a 2Xn-2) 


where S” denotes the variance of the X-sample and _X,, > 
X,—1 > Xn,—2 denote the three largest data values in 
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Figure 4. Efficiency of five estimators for the mean of CHICKEN. 


that sample. This estimator has a small bias in infinite 
populations. However the coverage of the standard con- 
fidence interval X, + Z,_9/2/v(X;) is often well below 
the nominal 100(1 — a)% level especially when the under- 
lying distribution is skewed. Further research is needed to 
obtain reliable confidence intervals for estimators of the 
mean of skewed populations. 
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APPENDIX 1 


Proof of Proposition 1 The assumption that Y is more 
skewed than X implies that there exists a convex function 
wW such that y(X) and Y have the same distribution. Let 
R denote R(Fy,n). To prove the result, it suffices to 
show that y(R) < R(Fy,n). This is equivalent to 


[1 — Fy(x)]dx.  (A.1) 


n— 1 


Aas a I40@) L \" 
Y(R) 
By Jensen’s inequality, E(Y) = E[W(X)] > Y[E(X)]. 


Thus using (2.3), the left hand side of (A.1) is less than or 
equal to 


R—E(X) ¥(R) — vIE(X)) 
R= E(X) 


R—-E(X) 
n—-1 


Y(R)= 


n—-1 


| ti tI avy aR) 


R 


where wW’ is the derivative of y. Since W’ is increasing, the 
left hand side of the above inequality is less than or equal to: 


Sead hea LAGay i 
Y(R) 


| (OA) ba ee deal WA Ch ae | 


R 


This shows that (A.1) holds. 
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Proof of Proposition2 The following result obtained by 
applying Theorems 2.7.5 and 2.7.11 of Galambos (1987) 
to the distribution F(z’*!) is used extensively. If the 
sample maxima of distribution F(x) converges to 
H3,9(x), then all the moments of F exist and 


OLY oa eee ees (A.2) 


\" [1 — F(x)] x? 
x g(x) 


where g(x) ~ A(x) means that g(x)/h(x) converges to 
1 as x goes to infinity. Using (A.2), R(F,n) is obtained 
by solving 


R-—p_1-— F(R) 
n-1 — g(R) 


Ciss20(1))): 


Let R = F~'(1 — a/n), then, up to (1 + o(1)), the 
above equation becomes 


ocala co 


Let dg = g{F-'(1 — 1/n)]F-'(1 — 1/n) anda, = g 
[F-'(1 — ao/n)|F~!(1 — ao/n). Since for large values 
of x,xg(x) is increasing, @) > a, and the solution to 
(A.3) belongs to the interval (a@;,@ 9). In order to prove 
the result, one has to show that a, /ay converges to 1 as 
n goes to oo. 


Since g(x) = f(x)/[1 — F(x)], one can write 


F-1!(1-1/n) 
a = exp| | ecoai| = 
F—1(1—-a/n) 


F-1(1-1/n) 
exp E — a —- | te" (odt], 


F—-1(1—ag/n) 


where the second expression is obtained by integrating 
by parts. Since tg’ (t)/g(t) is less then c, one has ay > 
exp(dg — @))do °. If a;/ao does not converge to 1, say 
a,/ay < 1 — € < 1 for an infinite sequence of sample 
sizes, the previous inequality implies that aj *° > exp(ao€). 
This is a contradiction since a, tends to o as n becomes 
large. The approximation for MSE(Xp) is obtained by 


using (A.2) with p = 2. 


Proof of Proposition3 Ifthe sample maxima of distri- 
bution F(x) converges to H, ,(x) then F satisfies the 
following properties (Feller 1971, p. 281): 


yer) lay = (A.4) 
- a lpia 


i" [1 — F(x) |x?*! 


for any p such that a — p — 1 = 0. By (A.4), R(F,n) 
is obtained by solving F(R) = 1 — [a —1+o0(1)]/n. 
This leads to the approximation for R(F,n). To derive the 
approximation for MSE (Xp), one applies (A.4) with p = 1. 
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Design Effects for Correlated (P, — P,) 


LESLIE KISH, MARTIN R. FRANKEL, VIJAY VERMA and NIKO KACIROTI! 


ABSTRACT 


We present empirical evidence from 14 surveys in six countries concerning the existence and magnitude of design 
effects (defts) for five designs of two major types. The first type concerns deft(p; — pj), the difference of two 
proportions from a polytomous variable of three or more categories. The second type uses Chi-square tests for 
differences from two samples. We find that for all variables in all designs deft(p; — pj) = [deft(p;) + deft (pj) }/2 
are good approximations. These are empirical results, and exceptions disprove the existence of mere analytical 
inequalities. These results hold despite great variations of defts between variables and also between categories of 
the same variables. They also show the need for sample survey treatment of survey data even for analytical statistics. 
Furthermore they permit useful approximations of deft(p; — p;) from more accessible deft (p;) values. 


KEY WORDS: Design effects; Survey sampling; Sampling errors. 


1. DESIGN EFFECTS FOR ANALYTICAL 
STATISTICS 


We explore the existence and the magnitudes of design 
effects for some special analytical statistics based on data 
from survey samples. The investigation is both method- 
ological and empirical, with data from several different 
surveys with different variables and from contrasting 
populations, hence subject to the risks of inconsistent 
empirical results. We often hear and read that probability 
sampling, while necessary for descriptive surveys, is not 
necessary for analytical surveys. In ‘‘Four Obstacles to 
Representation in Analytic Studies’’ one of us wrote that 
“Tn addition to those four real obstacles, we also encounter 
another, which is more artificial, in the denials of the need 
for representation’’ (Kish 1987, Section 2.7). Sampling 
investigations show that complex probability selections, 
especially clustered sampling, have no appreciable influence 
on descriptive statistics (like means and regression coef- 
ficients), but can have drastic effects on inferential sta- 
tistics, like confidence intervals, tests of significance (Kish 
and Frankel 1974). 

Design effects are defined as deft? = actual variance/ 
simple random variance of same n, both estimated. And 
values of deft > 1 have been shown for sampling errors 
not only of means, but also for analytical statistics like 
differences of means (and Chi square tests), regression 
coefficients efc. It is true that considerable reductions and 
differences of deft values have been found for some ana- 
lytical statistics. The differing deft values are not mere 
necessary mathematical consequences of the sample 
design, which may be deduced once for all. They have 


empirical content and therefore they need to be replicated 
with empirical investigations (Kish and Frankel 1974; 
Kish 1987, 7.1; Kish 1965, 14.1-14.2; Rao and Wu 1985; 
Scott and Holt 1982; Skinner, Holt, and Smith 1989). In 
this paper we investigate the possible effects and the 
magnitudes of design effects for a set of related statistics 
that have not been investigated before. On the contrary, 
in several statistical papers the absence of design effects 
was merely assumed by the authors (all justly famous), and 
apparently passed on by the journal referees, without 
warning the readers. We shall see if deft is reduced or 
eliminated for this set of analytical statistics (Cochran 
1950; Mosteller 1952; Scott and Seber 1983; Seber and 
Wild 1993). 

Furthermore, we also propose explicitly, as has been 
implied before, that the existence of considerable values 
of deft is strong evidence for the need for probability 
selections. It would be difficult to assume a model of 
a population distribution where the selection design 
was unimportant (or uninformative) but produced 
considerable design effects. The reverse does not hold: 
absence of design effects is necessary but not sufficient 
evidence for license to neglect probability selection. This 
proposition gives added importance to our study, which 
relates deft(p; — p;) for analytical statistics to deft (p,) 
and deft(p;) for two of several categories of the same 
variable. 

Section 2 describes the five related problems (designs) 
for which sampling errors are described in Section 3. 
Section 4 discusses the empirical evidence in the tables. 
Section 5 places our findings in the context of earlier work 
on defts for subclasses and their differences. 


' Leslie Kish, ISR, University of Michigan, Ann Arbor MI 48106, U.S.A.; Martin R. Frankel, NORC and City University of New York; Vijay Verma, 
University of Essex, Colchester, C04 3SQ, U.K.; Niko KaCciroti, Institute of Statistics, Tirana, Albania. 
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2. SIMILAR STATISTICS FOR 
FIVE DESIGNS 


It has been shown that five designs (problems), of two 
distinct types, can be treated with the same simple statistics 
(Kish 1965, Section 12.10). For our empirical and simple 
presentation we use symbols for sample values (like deft, 
p; and n;), even when occasionally capitals for population 
values would be more appropriate. 

The difference of proportions p» — Po = N2/n — No/n 
expresses the desired estimate, where n = my + ny + 
ny + ... Nz is the sample size, with n units selected and 
weighted equally. Furthermore, under simple random 
sampling assumptions, the variance of (p, — po) is 
(= fila Po (eo) na) 


Type A Comparisons 


1. The difference between two categories (”, — No)/ 
n = (p> — Po) of a polytomy can represent preference 
between two parties among several (xk) in voting 
surveys, or between two brands of automobiles in 
market research, or two of several attitudes, opinions, 
behaviors on one variable, etc. The other (k — 2) 
choices are summed into p, and disregarded in the 
difference. (Also treated by Scott and Seber 1983.) 


2. Rank values of —1, 0, +1 (or 0, 1, 2 orc, c + 1, 
c + 2) can be assigned to an ordered trichotomous 
variable without a metric, and viewed as a simple form 
of the difference of two categories. This form is partic- 
ularly useful for computations of sampling errors, 
because all the five designs can use —1, 0, +1 for 
instance as a transformed computing variable. 


3. The difference of proportions from two different vari- 
ables (x and y) may be treated as in (1) and (2). Define 
as positive in x (or success) only those elements that are 
positive in x but not in y, so that mg = n(X1, Yo). 
Similarly define as positive y the mo, = n(Xo, Y)- 
Then (79 — N0)/N = (Py — Py) is the net difference 
in the proportion of positives in x and y. Those that are 
positives or negatives in both x and y do not count in 
the differences. Thus we have a case of three categories 
as in (1) and (2). An example is the difference between 
the proportions who would ‘‘stop all nuclear testing,”’ 
and those who ‘‘want complete nuclear disarmament’’; 
or who would ‘“‘force Iraq to leave Kuwait’’ and who 
would ‘‘remove Saddam from power,’’ (Wild and Seber 
1993). However, the two categories may also come from 
two different surveys of the same n cases, as in a quality 
check, or from dual frame observations, or from two 
waves of a sample. These situations resemble those of 
(4) and (5). 
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Type B Comparisons 


4. Test-retest and before-after are terms for designs in 
which the same subjects undergo two observations. 
Then dichotomous answers ny = M9 denote the number 
of negative changes; 49 = Mo, the number of positive 
changes; and n,; + Moo the sum of the unchanged 
positives and negatives. Positive and negative answers 
are respectively denoted here as 1 and 0, and the first 
and second wave by the order of the subscript. The 
difference (M9 + 1;) — (Mo, + 111) = M10 — No = 
Ny — No measures the change between positives for 
the two observations; and py — Po = N2/n — No/n 
measures the change in proportions. (McNemar 1949; 
Cochran 1950; Mosteller 1952). 


5. Matched pairs of 1 pairs of subjects can also be treated 
as a generalization of the test-retest design (Mosteller 
1952). For example x pairs of randomized subjects may 
represent experimental versus control treatments; orn 
pairs of boys versus girls matched on control variables. 
The statistical treatment (p;9 — Po) of then pairs of 
matched subjects is the same as for the n pairs of 
treatments on the same n subjects (4). 


The similarity of statistical treatment for these five 
designs of two distinct types is convenient, and we present 
empirical results for both types. ‘‘It also has heuristic value 
that has been overlooked in recent publications (Scott and 
Seber 1983 and Wild and Seber 1993). The Chi-square test 
for types 4 and 5 was published early (McNemar 1949; 
Cochran 1950; Mosteller 1952), and the similarity to the 
categorical cases 1, 2, 3 was shown”’ (Kish 1965, 12.10). 
(Kish was wrong in denoting ‘‘trichotomies and matched 
dichotomies,’’ as ‘‘Trinomials and Matched Binomials,”’ 
which terms refer to IID samples only.) 

All of these deal with differences of proportions p; 
based on count variables n;. Extensions to correlated 
differences (y; — y;) for other variables are possible, but 
not within the scope of our study. Practical examples 
would include the difference in dollar shares (not only 
numbers n;) between two automobile makes from a total 
of > y; sales. 


3. SAMPLING ERRORS AND DESIGN 
EFFECTS 


For simple random samples of size n it can be easily 
shown (Kish 1965, 12.10) that 


var (pz — Po) = 


1L— 
Pa | [p2 + Po — (P2 — Po)? |/n. 
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Most of the examples found and shown come from large 
survey samples, where the (1 — /) can be disregarded. It 
is worth noting that for the element variance 


Pie DoD oa Po)? = P2492 + Pogo + 2P2P0; 


where the last term cov(p2,)9) = —P2Po represents the 
covariance arising because p> and po are competitive parts 
of the same sample, rather than proportions from inde- 
pendent samples. The difference of proportions squared 
(p> — Po)” will usually be a small correction term, 
and without it we have the equivalent of the variance 
(Pp. + Po)/n of two independent Poisson samples. Fur- 
thermore, note that (Kish 1965, 12.10): 


The Chi-square test has been applied to some of these 
problems, treated separately (Cochran 1950; Mosteller 
1952; McNemar 1962, p. 225). This is essentially 
(n> — No)*/(ny + No) the square of the difference 
divided by its variance, under the null hypothesis 
Ny = No. It applies the exact theories available for 
tests of null hypotheses in small samples, including 
the “‘Yates correction,”’ all based on the assumption 
of simple random sampling. However, there are great 
advantages in treating these problems in large samples 
as estimated means with proper standard errors. First, 
instead of being confined to testing null hypotheses, 
we can make inferences with the probability intervals 
(D2 — Po) + t, Se (P2 — Po). Second, the formulas 
for standard errors of complex samples can be applied 
directly to the mean (p>. — Po). Third, the logical 
structure of this statistic (p. — Po) can be seen more 
clearly in its application to several distinct problems. 


Correlated proportions originate usually in data from 
complex surveys, and the computations of variance should 
be appropriate to the sample design. The variance formulas 
for stratified complex samples can be adopted, but the 
direct formula has eight terms (Kish 1965, 12.10.3). 
Instead, it is convenient to translate the problem into a 
trichotomous variable, with values of —1, 0, +1 as in 
design 2 of Section 2; and the computations of Section 4 
used that translation. 

Then comparisons between variables and between 
samples can be facilitated by recourse to the design effects: 


computed variance of (p2 — Po) 
[p2 + Po — (P2 — Po)*]/n 


deft?(p, — po) = 


A few words are needed about limitations on the use 
of deft as a tool for robust approximations. They serve 
well for clustered and multi-stage samples using ultimate 
clusters (primary selections) for computing sampling 
errors. However, we avoided the problem of weighted 
samples, because their treatment would be too specific and 
perhaps too complex. Weighting for nonresponse would 
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not be important for the ratio of deft(p; — p;) to deft(p;). 
However weights for gross inequalities of selection proba- 
bilities need specific treatments. Nevertheless, inference 
and experience indicate that deft values are less affected 
by weights than are the variances and means themselves. 
Furthermore we conjecture that the relations we found 
between the values of deft(p; — p;) and deft(p;) will 
hold also for weighted data, if these are not extreme or 
pathological. 

An approximate but dependable relation of deft (p; — p;) 
to deft(p;) and deft(p;) would be useful to allow infer- 
ences from the latter, which are routinely and easily 
computed, to the former that are not. Several alternative 
conjectures may seem reasonable, and none can be math- 
ematically derived, nor excluded. 


1. Deft(p; — p;) = 1 of no design effect was assumed 
implicitly in the five publications referenced in Section 1. 


2. Deft(p;) > deft(p; — pj) > 1 denotes persisting but 
lower effects than for the deft(p;) for proportions. 
This happens for ‘‘crossclasses’’ and their comparisons 
(Kish 1987, 7.1). This also seemed reasonable to several 
experienced statisticians we polled. 


3. Deft(p; — pj) = [deft(p;) + deft(p;)]/2 is what 
we actually found to be a good approximation for all 
of our data, from different populations and designs. 
This conjecture seems reasonable, because design effects 
due to clustering for individual p; can apply similarly 
to the variable created from the difference (p; — pj) 
of two of them. 


4. Inconsistent results would have been possible, but 
annoying by preventing inference. 


4. EMPIRICAL RESULTS FOR Deft(P; — P;) 


Without strong theoretical or mathematical basis 
for favoring any of the four alternative conjectures, 
empirical results about deft(p; — p;) become essential, 
linking these to the computed values for deft (p;). These 
resemble our more familiar conjectures about deft(p;) = 
Jl + roh[b —1]; their value depends on several factors 
that affect roh, the coefficient of intraclass correlation, 
in addition to the average cluster size b (Kish 1965, 5.4, 
8.2). The values of deft (p;) vary greatly between surveys, 
also between variables for the same survey (Kish, Groves 
and Krotki 1976; Verma, Scott and O’Muircheartaigh 
1980; Verma and Lé 1995). However, survey statisticians 
gain knowledge from empirical investigations of sampling 
errors from diverse surveys, which also permit relating the 
deft values of complex statistics to the simpler deft (p;) 
(Kish L. 1995; Rao and Wu 1985; Rao and Scott 1987). 
Similarly, to learn about the relation of deft(p; — p,) 
to deft(p;) we have here empirical results from many 
variables and from many surveys. 
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In this first essay into this field we present data from 
fourteen surveys, which represent a great variety of situa- 
tions. Eleven surveys presented as 5 sets of results (Figures 1 
and 2 and Tables 1-3) deal with paired differences of 
categories from single surveys (Type A). Three sets of 
results (Tables 1-3) come from social surveys, followed by 
two sets (Figures 1 and 2) from the Demographic and 
Health Surveys on population data. Finally three other sets, 
each dealing with two waves of data, each based on two 
reinterviews with the same respondents (Tables 4, 5 and 6), 
represent type B designs of comparisons. 


Tables: 


1. The National Election Study of 1986 of the Institute 
for Social Research of the University of Michigan, 
N= 13). 

2. The National Education Longitudinal Study (NELS) 
of 1988, the National Opinion Research Center of the 
University of Chicago, n = 24,355. 

3. The National Longitudinal Study of Labor Market 
Experience of Youth, conducted by the National Opinion 
Research Center of the University of Chicago, n = 5,857. 


4. National Election Studies Panels 1990 and 1992, Survey 
Research Center, Institute for Social Research, Ann 
Arbor, MI 48106. 


5. Panel Study of Income Dynamics 1983 and 1987, 
Survey Research Center. 


6. Americans’ Changing Lives 1986 and 1989, Survey 
Research Center. 


Figures: 


1. Demographic and Health Surveys of Morocco, Niger, 
and Colombia, MACRO International. 


2. Population Census of Indonesia, Rural Java strata 
(unpublished data). 


We note the following important, useful, EMPIRICAL 
results. 


1) First and foremost: The design effects deft (p; — p;) for 
the differences are usually NO LESS than the deft (p;) 
for the proportions themselves, and deft (p; — p;) = 0.5 
[deft (p;) + deft(p;)] approximately in all cases. They 
vary together, along with the considerable variation for 
deft values between variables, and also with the lesser 
variation between pairs of categories for the same 
variables. Researchers who neglect deft commit the usual 
under-statement of sampling errors for statistics from 
clustered surveys. This observation is not only interesting 
but also a useful model for inference, because the other 
three sources of variation — across variables, categories 
within variables, and sampling errors of individual 
statistics - are all greater. 


2) We can find these results in all the 14 sets of survey data 
in the tables and graphs, and we can illustrate them now 
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with Table 1. Note that defts vary from essentially 1.00 
for variable D (problems in country) to as high as 2.32 
in variable A (religion) which implies deft? = 2.327 = 
5.38. That our empirical rule (1) holds over the range 
is reassuring. Such variation between variables in the 
same sample are common and should force us to 
abandon the practice of using a common average for 
all defts of a sample (Verma and Lé 1995; Kish 1995). 


Furthermore, we emphasize here the great variation in 
deft values for the five categories of the same variable 
from 1.21 to 2.32 (No. 3 for ‘‘fundamental’’ protes- 
tants). It follows that deft(p; — p;) is large only when 
i or j is category 3 for this variable. These variations 
among the defts for categories of the same variable 
mean that they should be computed for all categories 
rather than for only a single ‘‘representative’’ category. 
These large possible variations between categories of 
the same variable are an important new finding in our 
results, that seems to have escaped notice before. 


3) There are also sampling errors in the computed values 
of the defts. Only statisticians who have computed 
many sampling errors and design effects seem to get the 
‘*feel’’ for how great these can be. They may be mostly 
responsible for the few cases where deft(p; — p;) 
fails to fall between deft (p,) and deft(p;) and either 
deft(p;) < deft(p; — pj) > deft(p;) or deft(p;) > 
deft(p; — p;) < deft(p;). Incidentally, these cases 
also show that our results are not mathematical conse- 
quences, but empirically based. 


The empirical results presented in Figures 1 and 2 
further confirm the findings already presented in Tables 1, 
2;.and 3. Here: also. we see that: 1) deft(p; — p))\= 
[deft(p;) + deft(p;)]/2 approximately, along the 45° 
line; that 2) those equalities hold along a wide range of 
designs effects; and that 3) the variation between variables 
is large indeed. This large variation is particularly evident 
for rural Indonesia, with deft values over 4, hence deft? 
values over 16. These large clustering effects are due to the 
large cluster sizes: with b = 133 and 137, the values of 
roh = 0.12 are enough for large defts. Note that these 
empirical results come both from very diverse populations 
and diverse variables; different from each other and from 
the data of Tables 1, 2, and 3. Figure 1 has data from 
3 countries (Morocco, Niger and Colombia) hence 6 pop- 
ulations, because the urban and rural defts are quite 
different. Figure 2 shows results for males and females 
who are quite distinct populations for the occupational 
variables, though less so for the educational classes. 

The empirical data in the tables of studies 4, 5, and 6 
were awaited with doubt and anxiety. True that the pre- 
ceding five sets resulted in similar conclusions, although 
they dealt with eleven different populations and scores and 
variables. But studies | to 5 all dealt with pairs of categories 
from polytomies, designs 1 and 2 of Type A. But now we 
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Subclasses according to current use of contraception 
(DHS surveys) 
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Figure 1. Comparison of deft (pj; — p;) to the average of 
deft (p;),deft(p;) for categories by current use of 
contraception*. Illustration of six populations from 
Demographic and Health Surveys. 
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Subclasses by occupation and level of education 
(Indonesia, rural) 
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@ Male, Occupational classes 
4 Female, Occupational classes 
o Male, Educational subclasses 
as a Female, Educational classes 


deft of DIFFERENCE between pair of subclasses 


0.0 05 1.0 AS 20 25 3.0 3.5 40 45 5.0 
AVERAGE of subciass defts in the pair 


Figure 2. Comparison of deft (p; — D;) to the average of 
deft (p;) ,deft(p;) for categories by occupation and 
level of education by sex. Illustration from a population 
census. 


* | = not using any method of contraception 
2 = using only traditional method 
3 = using a modern ‘reversible’ method 
4 = sterilised. 
Table 1 
The National Election Study of 1986 of the I.S.R. of the University of Michigan (n = 2,135) 
Categories Defts for Categories Defts for 
i-J P; P; Average (P7— 8) IJ P; P; Average (Pe 2:) 
A. Religion B. Abortions Beliefs 
1-2 1.21 1.42 e382 1.10 1-2 E27 Sy) 1.12 207] 
1-3 PAL 232 ont 2.02 1-3 eT 1.28 1.28 1.32 
1-4 1) 1.50 1.36 1.18 1-4 1.27 1.31 1529 1.36 
1-5 141 1.18 me ily 2-3 SH 1.28 Li 1.08 
2-3 1.42 D3) 1.87 E93 2-4 Sil eS 1.14 1.16 
2-4 1.42 1.50 1.46 LoS7 3-4 1.28 esi 1.30 11,32 
2-5 1.42 1.18 1.30 Ly Mean HSI. 1.24 Dea 1.20 
3-4 232 1.50 1.91 2.03 
3-5 Des) 1.18 ATS 2.04 D. Problems in Country 
4-5 1.50 1.18 1.34 [EL 
1-2 1.07 94 1.00 .98 
Mean 1.56 1253, 1.54 ESS) 13 1.07 1.04 1.05 1.09 
1-4 1.07 293 1.00 ale, 
C. Support Reagan 23 94 1.04 99 1.01 
1-2 132 1.10 Lil 1.07 2-4 94 .93 .93 85 
1-3 es 2 .86 1.09 1.26 3-4 1.04 .93 .98 .82 
1-4 1252) 1.48 1.40 1.50 Mean 1.02 .97 99 98 
2-3 1.10 .86 98 .96 
2-4 1.10 1.48 1.29 1.38 
3-4 86 1.48 LT, 1.09 
Mean HME eon 1.19 HPI 
Overall mean 1.23 1.24 E28 1.24 
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Table 2 


The National Education Longitudinal Study (NELS) of 1988, the National Opinion Research Center 
of the University of Chicago, (n = 24,355) 


Categories Defts for Categories Defts for 
= i ey tag | Average Ue 249) = if P; P; Average Pi Ep 
A. Education Status B. Classes are boring 
1-2 1388 122 1.30 ites lat 1-2 .99 Lea 1.05 1.04 
1-3 1.38 1.14 1.26 1.16 1-3 .99 el2 1.06 1.07 
1-4 1.38 1.19 29 1.30 2-3 tail WA Tat ens 
1-5 1.38 1.42 1.40 1.54 Mean 1.03 HIE 1.08 
- : : 18 ell : 
mi oe a 4s 1.24 C. Freedom to pursue interest 
2-5 2 eee 1.32 1.45 We sion ioe Wet 
3-4 Lig ic19 ie 1.18 1-3 S28, FOF HELOS Is Us 
3-5 er ay ee 1.28 37 ue palOne aai28 Wee: a 
4-5 1.19 1.42 eu 1.20 Mean eZ2, 1.09 HS) JTS) 
Mean LZ, 1.28 WPA S723) D. School offers good jobs 
E. Religion 1-2 1.24 1.07 1.16 ONY 
12 248 ee 83 2.65 2.74 1-3 1.24 111 1.18 
133 2.48 2.02 2.25 2.09 2-3 1207 iil 1.09 1.01 
D3 2.83 2.02 DAD 2.59 Mean 1.18 1.10 1.14 1.14 
Mean 2.60 2.29 2.44 2.47 F. Dad education 
I. Feel good about self 1-2 1.61 1.76 1.69 1.83 
12 142 28 1.35 1.37 1-3 1.61, 1.68 1.65 1.69 
2-3 1.76 1.68 72 2.48 
Mean 1.65 NA 1.69 1.99 
Overall mean 1.48 1.4] 1.45 1.49 
Table 3 
The National Longitudinal of Labor Market Experience of Youth, Conducted by the National Opinion Research Center 
of the University of Chicago, (n = 5,857) 
Categories Defts for Categories Defts for 
=) Je P; Average (Pp Py) =i P; P; Average (P)-—-P)) 
A. Chance is important in my life B. Something stops me 
1-2 1.26 1.20 23 1.06 1-2 1.07 ea? 1.14 1.04 
1-3 1.26 1.18 Lay 1.30 1-3 1.07 ee 1.10 1.14 
1-4 1.26 1.16 21 1.28 1-4 1.07 1.09 1.08 1.09 
2-3 120) 1.18 1.19 1222 2-3 La? 1A I gil! 1.28 
2-4 1.20 1.16 1.18 1 255 2-4 p22 1.09 1.16 1.14 
3-4 1.18 1.16 il 1.05 3-4 1122 1.09 ll 1.07 
Mean Tp23, HoH 1.20 1.19 Mean ers HIP He Jts3 Jels 
C. Have control of my life D. I am as worthy as others 
1-2 ists) 1.06 1.10 1.09 1-2 ih i7h i113} HS) 1.16 
1-3 ats 1.10 all? ans 1-3 1.17 1.07 Lee 1.16 
2-3 1.06 1.10 1.08 1.07 2-3 1s 1.07 1.10 1.08 
Mean TT 1.09 1.10 Ned Mean JG 1.09 ale IS 
E. Plans hardly work out F. I am satisfied 
1-2 1.19 1.07 sil} iT 2 1-2 1.19 I ll? 1.16 1.16 
1-3 1.19 13 1.16 1.20 1-3 1.19 its: 1.16 1.20 
2-3 : as) LO 1.08 2-3 ae es oils 1.09 
Mean 1.15 oA Tes 1.13 Mean WH eS, ISIS HIS 
I. Mother’s work 
1-2 1.49 1.36 1.43 1.41 
1-3 1.49 1.52 i651 153 
2-3 1.36 es 1.44 1.44 
Mean 1.45 1.47 1.46 1.47 


Overall mean 1.20 LAG 1.18 1.19 
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Table 4 


National Election Studies Panels 1990 and 1992, 
Survey Research Center, Institute for Social Research, 


Ann Arbor 

Categories Defts for 
Strongly approve Bush Nl4e93 1.04 1.02 
Approve Bush foreign 

policy ye NOS 299 1.00 
Strongly disapprove Bush 

foreign policy e235 1e24! 1.24 iL 53 
Approve Bush economy OT m4 .96 .96 
Strongly approve Bush 

economy 1.14 1.04 1.09 1.10 
Approve Bush 1.00 1.00 1.00 1.00 
Strongly disapprove Bush 1.16 1.10 iets} othe 
Watch campaign on TV ase) Jays) 122 1.40 
Mean ILO TID 1.08 UT 

Table 5 


Panel Study of Income Dynamics, 1983 and 1987, 


Survey Research Center, Ann Arbor 


Categories* Defts for 
before/after (83/87) P; P; Average (P; — Pj) 
Live in South 122 1e25 1.23 1.11 
Age of head of family 1228) 1233 1331 1.37 
Family size 129 e143 1.36 1.47 
Number of children 

in family 3) 148} 11338} 1.49 
Work hours of head 1.12 @.84 .98 1.03 
Age of youngest child 93) 2 9i E92 .87 
Mean WIS AG 1.19 1.22 


* All variables are categorized in two categories. 
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sought data for Type B comparisons from panel surveys, 
so that we could investigate the conjectures for the test/ 
retest and before/after experimental designs. Mathemat- 
ically these can be easily shown to resemble polytomies 
(i.e., tetratomies), but from that to the empirical values 
of design effects leads through a ‘‘black box.’’ Hence these 
empirical values are so much more valuable and remark- 
able. Here we found considerable design effects for Chi 
square tests for analytical comparisons. 


5. PRESENT FINDINGS IN THE CONTEXT 
OF RELATED RESEARCH 


A great deal of empirical information is available from 
previous work by the authors and by others on design 
effects for the total sample, for subclasses, and for differ- 
ences, for diverse variables and designs. It would be useful 
to put the present findings in the context of that work. 

It has been found that nature of the survey variables 
being estimated is a major (often the main) determinant 
of the magnitude of the design effects: vastly differing 
defts can occur for different types of variables even with 
the same samples or with similar designs. For this reason 
we have always recommended that defts be computed for 
many different variables, while it is generally less important 
to compute them for many different subclasses, especially 
for different categories of subclasses defined in terms of 
the same characteristic. 

The present findings illustrate that defts can differ 
greatly also among different categories of the same survey 
variable, estimated with the total sample as the common 
base. Therefore each individual category and each differ- 
ence between pairs of categories, even when defined in 


Table 6 
Americans’ Changing Lives, 1986 and 1989, Survey Research Center, Ann Arbor 

Categories Defts for Categories Defts for 
before/after P; P; Average (P; — Pj) before/after P; P; Average (P; — Pj) 

A. Get together with friends B. How often do you exercise 
Once a week 1.30 1.26 1.28 1.28 Often ilesyil Jhates7/ (oe) 1.26 
2-3 a month .88 1.00 .94 1.02 Never 1.62 E97) 1.80 1.41 
Mean 1.09 1.13 LT ES) Mean 1.56 1.82 We AD 1.34 

C. How Satisfy Are You D. How do you like your home 

Very satisfy 1.28 7) 1725) E33. Very much 1.24 .90 ON) 91 
Not satisfy 1.04 1.16 LiKe 1.00 Not much 1.33 .98 1.16 iba JU 
Mean 1.16 RLS: 1.18 yf Mean AG 94 HM 1.02 

E. How often work in garden F. I have a positive attitude 
Often 1.40 We Ke) 1.28 1.19 Agree 1.10 33 122 1.19 
Rarely 91 Jil 1.01 1.18 Disagree 1.05 1.28 Ibsen ill 
Never 1.66 Nel V9 1.42 1.26 Mean 1.08 Hsp 1.20 1.20 
Mean des. Hels) 1.24 Isai 
Overall mean Ngpeas HPXe) Ie 1.18 
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terms of the same survey variable, needs to be regarded, 
in a sense, as a separate variable in its own right for the 
purpose of computing and analyzing design effects. 

As to the relationship between defts for subclasses and 
subclass differences, previous research has mostly dealt 
with the following situation. With the total sample 7 parti- 
tioned into subclasses i of size n; = p;.n, deft(r;) values 
for statistics r; (such as a proportion m;/n;, mean Y y;/n;, 
ratio ) y;/ ¥.x;), estimated over subclass elements n; as the 
base, are related to deft (7) for the same variable estimated 
with the total sample as the base. Similarly, deft(7; — r;) 
for subclass differences are related to deft(r;),deft (7; ) 
based on individual subclasses and to deft (r) based on the 
total sample. Numerous computations confirm these rela- 
tionships to be in accord with our conjecture (2) of section 3: 


deft (7) > deft(77)s-andsdethng,)"> dett (7°17; >? 


These effects of covariances on design effects of clus- 
tered samples are essentially empirical (even sociological 
in a broad sense); and they must be so verified. 

Similarly with our newly discovered relationship for 
(p; — p;) for two categories, which are so different from 
the above. The relations deft(p; — p;) = [deft(p;) + 
deft (p;) ]/2 are also empirical and approximate and they 
must be verified over and over again. But they seem to be 
widely applicable in our data, and clearly better than the 
other assumptions, such as deft(p; — p;) = 1 that have 
been often assumed until now. 
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Alternative Adjustments Where There Are Several Levels 
of Auxiliary Information 


F. DUPONT! 


ABSTRACT 


Regression estimation and its generalization, calibration estimation, introduced by Deville and Sarndal in 1993, 
serves to reduce a@ posteriori the variance of the estimators through the use of auxiliary information. In sample surveys, 
there is often useable supplementary information that is distributed according to a complex schema, especially where 
the sampling is realized in several phases. An adaptation of regression estimation was proposed along with its variants 
in the framework of two-phase sampling by Sarndal and Swensson in 1987. This article seeks to examine alternative 
estimation strategies according to two alternative configurations for auxiliary information. It will do so by linking 
the two possible approaches to the problem: use of a regression model and calibration estimation. 


KEY WORDS: Auxiliary information; Regression estimator; Calibration estimator; Two-phase sampling. 


1. INTRODUCTION 


Using the regression estimator studied by Fuller (1975), 
Cassel, Sarndal and Wretman (1976), Sarndal (1980), 
Gourieroux (1981), Isaki and Fuller (1982), and Wright 
(1983), it is possible to improve a posteriori — that is, after 
the sampling has been completed - the estimate of a total 
of a variable of interest on the basis of auxiliary variables 
X,, ...,X, for which additional information is available. 
The variance in relation to the Horwitz-Thompson esti- 
mator is reduced by using the regression estimator, provided 
that one knows the true value of the target population 
totals of the auxiliary variables, which will constitute the 
additional information referred to as auxiliary informa- 
tion. Deville and Sarndal in 1992 proposed a class of esti- 
mators derived from a reweighting approach that addresses 
the same issue of variance reduction: calibration esti- 
mators. By calibrating sampling weights it is possible to 
incorporate a posteriori auxiliary information of the type 
totals X,, ..., X, of k variables x,, ..., x, into the esti- 
mate made on the basis of the new weightings and thus to 
improve the estimate. This approach generalizes regression 
estimation, which is one of the elements of the class. 

However, in surveys based on sampling, there is often 
usable additional information that is distributed according 
to a more complex schema than what has been described 
above, especially when the sampling is carried out in 
several phases. This article looks at different strategies for 
using this complex auxiliary information in the framework 
of two-phase sampling, with the possibility of generalizing 
to more than two phases. 

When the sampling plan entails two phases, the aux- 
iliary information consists of information known for the 
entire population, but also of information known for the 


sample resulting from the first sampling phase. These two 
bodies of information may concern different variables. 

In their 1987 article, Sarndal and Swensson propose an 
estimator that uses all the auxiliary information available 
for a two-phase sampling, with different auxiliary infor- 
mation for the total population and the population obtained 
from the first-phase sampling. This is an estimator adapting 
the principle of the regression estimator when the infor- 
mation known for the individuals obtained from the first- 
phase sampling is considered to be substitutable for the 
aggregated information and to be of better quality than 
the information available for the target population as a 
whole, for purposes of estimating the variable of interest. 
However, in practice it often happens that these two bodies 
of information are complementary rather than 
substitutable. We have thus sought in this study to develop 
the regression estimate in a context in which the bodies of 
auxiliary information are complementary. 

Furthermore, insofar as calibration estimation gener- 
alizes regression estimation when the auxiliary information 
is at only one level, we have sought to adapt calibration 
estimation to this context. We review the various cali- 
bration strategies in order to propose the most suitable 
ones, seeking to relate them to generalizations of regression 
estimation that are possible in this context. 

We show (Section 2) that the joint use of two different 
bodies of auxiliary information leads to two regression 
models and three associated decompositions of the variable 
of interest. The regression model assisted approach (RMAA) 
thus enables us to derive 3 alternative estimators. 

In turn, the calibration approach (CA) (Section 3) 
enables us to derive 4 estimators. Each of these estimators 
may be related to (associated with) the three estimators 
derived from the regression model approach. 
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Thus (Section 4), the two approaches may be linked 
together and result in three classes of estimators, each 
associated with a decomposition of the variable of interest. 
The estimators of a given class have the same asymptotic 
variance. 

When strategies are evaluated on the basis of the 
sampling plan alone, our choice is directed toward the 
third class of estimators, which is superior to the other two 
from the standpoint of variance. 

When strategies are evaluated on the basis of a modelling 
of the variable of interest, the preferable class of esti- 
mators is the one associated with the modelling adopted. 

In a situation in which we wish to adjust a survey, and 
in which we wish simultaneously to correct the biases that 
would result from the use of gross weightings and to 
reduce the variance, the findings must be adapted: the 
changes introduced in the weightings to correct the biases 
are greater than the corrections for variance reduction. 
Hence the variables will be incorporated into the calibration 
once it appears that they are affecting the probability of 
selection and thus participating in the creation of the bias. 

When the auxiliary variables are qualitative, the choice 
between a priori and a posteriori use of the auxiliary infor- 
mation - that is, between its use at the sampling stage and 
at the adjustment stage - still rests on the distinction 
between the two modellings of the variable of interest. 

These findings may be extended to samplings of more 
than two phases. 


2. NOTATIONS 


The framework is that of a two-phase sampling. Assume 
that auxiliary information is available at two different 
levels: the target population and the population obtained 
from the first-phase sampling. The situation may be 
diagrammed as follows: 


Size Vector of available auxiliary 
variables 
N x] 
Nq x1 x2 
n xX; xX2 


where U represents the target population for which the 
values of the vector of variable x, are known or, failing 
that, the total X; = VY jc¢y Xj. Sg represents an interme- 
diate level of sampling for which the values of the vectors 
of k, variables x, and k» variables x) are known for all 
individuals. We denote as z;, the probability of selection 


from the sample associated with the first phase of the 
sampling. s represents the final sample for which are 
available the values of the variable y, the total of which 
we are trying to estimate, as well as the values of the 
vectors of the auxiliary variables x, and x7. This is denoted 
ast; = P(h| Sz)e 

We hope to make optimum use of all this auxiliary 
information in order to improve the estimates that will be 
made on the basis of the data gathered from the sample 
that results from the second sampling phase s. 

An obvious first idea is to try to generalize the regression 
estimator in this context. 


3. REGRESSION ESTIMATION APPROACH 


3.1 The Information Contained in x, is Considered to 
be Substitutable for the Information Contained in 
x, for Estimating y and to be of Lesser Quality 


In their work, Sarndal, Swensson and Wretman propose 
the following regression estimator for estimating the total 
of y: 


2-3 2h (ye - yee) ee 


(pia 
i€s Ee ae 


( yy uOr eres aah a 


i¢U i€sg gt 


where the second term is the correction for poor estimation 
on s, and the third is the correction for poor estimation 
on Ss. 


The estimation can also be written: 
A : XDs = XB ) 
Y, = YY x/ b, se 3 ( ppsAl i2 42 mh 


3 : Tai 
ieU 1€Sq qi 


x (yj; — Xip by) 
Tj Nai 


i€s 


where the second term is the correction for poor approx- 
imation of y; by x, b; and the third is the correction for 
poor approximation of y; by x; b2; 


with b, re y kay as y Xi Yi 
T gi . Tj Tai 


1€S 


By b, =( y ea =f y =e) 
P Tj TN gj : Tj gi 


1€S 
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The underlying idea is that we have two concurrent 
models for y, namely: 


(1) ¥ = x40, + uy with E(u) =0 and V(uj) = of 
and 


(2) ¥j = Xigby + uj with E(uj.) =0 and V(uj) = of 


the second of which we believe is a priori better for pre- 
dicting the value of y;. Thus in this model-based perspec- 
tive, x, functions as a proxy of x,. A situation of this type 
corresponds, for example, to a case in which x, represents 
the update - that is, the update to the date of the survey - 
of the variable retrieved from the x, sampling frame. In 
other words, if x, were available at the level of the entire 
population, the estimator used would be 


Se xb, 4 yy (Wi = Xi 52) Xia br) 


1; 
ieU i€s pai 


Let us now imagine the case of a two-phase sampling 
survey of households. Assume that the sampling frame is 
made up of dwellings for which we have information 
consisting of dwelling size, denoted as x,, which is therefore 
known for all individuals in the target population. If all 
the individuals obtained from the first sampling phase are 
questioned on the composition of the household, denoted 
aS X, in particular on the number of children in the 
household, the two bodies of information appear to be 
complementary rather than substitutable for purposes of 
studying the household budget. This is further reinforced 
if instead of household composition, the information 
collected is the age or occupation of the head of household. 

In a model-based perspective, the alternative situation, 
in which the information contained in x, is considered 
complementary to that contained in x, for estimating y, 
thus naturally suggests itself. 


3.2 The Information Contained in x, is Considered 
to be Complementary to the Information 
Contained in x, for Estimating y 

3.2.1 Decomposition y; = xj,a, + Xj2a, + U; 

The underlying model is then: 


y, =Xf4a, + xa, + u; with E(u;) =0 and V(u;) = oF. 


The estimator to be used is then: 


y. = et oe eee motes +3 (Yi = 41 —%i242) 
Ze ni] 


T; Wai 
i¢U 1€Sq i€s beat 


with: 


Wer 


4 / 
> XiiXi Xi Xi2 \ 1 Xi i 
Merial UGG Tai TK; 
ay i€s Es i€s oe i€s Cue 
a = , , 
a Xj2Xi1 Xj2Xi2 X29 
Tai Nj Taji Mj Nai Mj 


i€s i€s i€s 

The variable here is broken down into three components 
Yi = XAG, + Xin, + ti;. The total of y is thus broken 
down into three components, each of which is estimated 
at the highest level, that is, with the greatest precision 


possible: 


-— U for xj a 6 
- S, for x/2@), and 
- s for ii;. 


3.2.2 Decomposition y; = xj¢, + M,,(%j2)'e2 + uj 


If we wish to make maximum use of the information 
contained in x, available on U, it is natural to introduce 
another formulation of the same model y; = xja, + 
X/,a_ + u; which isolates everything which in y can be 
taken into account through x,. It is written as follows: 


Vi = X11, + My, (X12) "Co + u; with 
E(u) =0 and, V(u). = of, 
where M,, represents the orthogonal projection, in the 
metric associated with the weights 1/7z,;, on the orthogonal 


of the vector space generated in s, (similar to #”) by the 
group of variables x,. 


M,,, (jz) is defined by: 
, / —] 
; Xj2Xi1 Xi Xi1 
M,, (Xj2) sezeimule)s ag ba) labaoeieas ) Xj + 
9 Tai . Tai 
1€Sq 1€Sq 
The associated natural estimator is then: 


“Se 


Yoaey c e 


i¢U 1€Sq 


J. 
xj,€, — My, Xi2€2) 


Ones 
Ye ; 


Tj WN agj 
jes yey: 


where ¢ = (6!) is the regression coefficient y = x’c; + 
(M,,X2)'Co + u estimated over s with weights 1/7,;77; 
(which differs slightly from (2 1) 


3.3. The Three Estimators Derived from the Model- 
based Approach 


The modelling approach has enabled us to construct 
3 estimators that can be rewritten synthetically by introducing 


128 Dupont: Alternative Adjustments Where There Are Several Levels of Auxiliary Information 


new notations. Throughout what follows, for a vector of 
a given variable z, the following notation will be used: 


» 
ll 
| - 
N 


N> 
T 
M4 
| 
AN 


With these notations, the three estimators are rewritten 
as follows: 

Y, = [X{b\] + [X3b, — Xb) + [LY — X3 bo) 
associated with the models: 


(1) y = x4b) + ui 
and 
(2) yj = Xj2b2 + Un 


Y, = [X{4,] + [X%3@] + [Y — X14, - X3@] 


associated with the model 


Yi = XQy + XjgQ_ + U; 


Y; = [X{@] + (M,,X36] + [Y — X14, — M,, X36] 
associated with the model 


Vie = Xie, + My, (X42) "Co + Uj. 


In the same manner as the regression estimator is gener- 
alized by calibration estimators, the problem of the use of 
auxiliary information at several levels may be dealt with 
through calibration theory, be attempting to construct 
calibration strategies adapted to the auxiliary information 
configuration examined in this article. 


4. CALIBRATION APPROACH 


4.1 Different Strategies Possible 


When we try to generalize the calibration estimate 
proposed in a context in which auxiliary information is 
present at a single level - that of the entire population - 
several strategies naturally suggest themselves: 


Strategy 1 


a) calibrate the structure of the Ist-phase sample s, on 
that of the total population U in terms of variable x,, 
then, 


b) calibrate the structure of the 2nd-phase sample s on 
that of the 1st phase sample s, in terms of variable x. 


Note: For the latter operation, it is better to take account 
of the preceding calibration in terms of x, in order 
to determine the reference value in the calibration 
in terms of x, on s,. If the preceding calibration is 
not taken into account, only the estimates made at 
the level of s, will benefit from the improvement 
made by stage a. A good way to convince oneself 
of this is to consider the specific extreme case where 
XxX, = X>- 

This strategy corresponds to the following calibration 
equations: 


Stage a: 
F(x,B1) 
ye a pDyssis = X 
i€sg Tai ieU 
which determines 6, then 


Stage b: 


F(x; AED ¢ 
3 FEU rae a icon 3 SUED eg = 8 


Taji Tj 3 Taji 
i€s Ress! 1€Sq as 


which determines £,, 


where F designates, as throughout this article, the function 
which is used in the calibration and which may be linear, 
exponential, truncated linear or logit (see Deville, Sarndal 
1993). 


Strategy 2 


Calibrate the structure of the 2nd-phase sample s simul- 
taneously in terms of variables x; and x, that is, 


- on the structure of the total population U as regards x, 
— on the structure of s, for x>. 


This second strategy leads us to the following calibration 
equations: 


F(xj,a; + Xa 
iS (Xi 1 i2 2) x1 = YS riinaicts and 


Tj Wj 
i€s at i¢U 


F (xj, + Xj2e2) =) X20 x 
Ds Sidon) pawn Ae 
Nail; 
i€s te i€Sq Tai 


which determines a, and ap. 


The first strategy offers the advantage of correcting the 
Ist phase weightings, that is, of incorporating the auxiliary 
information at the highest level. The second strategy, for 
its part, makes it possible to correct the weightings that 
will actually be used in the estimation, and in particular 
to obtain a perfect estimate of the total of x,. 
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A third strategy may be proposed; it combines the 
advantages of the above two strategies and would there- 
fore seem preferable to them: 


Strategy 3 


a) calibrate the structure of the Ist phase sample s, on that 
of the total population U in terms of variable x, , then 


b) calibrate the structure of the 2nd phase sample s simul- 
taneously in terms of variables x, and x, that is, 


- on the structure of the total population U as 
regards x, 

- on the structure of s, modified by taking account of 
the preceding calibration for x>. 


This strategy leads to the following calibration equations: 
Stage a: 
F(x 81) 
ys a ana yy Xi 
iésq a i€U 
which determines 6, then 


Stage b: 


F(x; + Xx; 
.P (Xi71 i272) rae x eRe and 


E Taj Mj ; 
i€s Gi! i¢U 


F(xy + Xx; F(x; m 
¥ (Xi71 i272) nS P ( ue ee oe 


3 Taji Tj : TT, 
ies ae i€sg oo 


which determines y, and 72. 


Lastly, a fourth strategy may be proposed; it may be 
seen as a variant of the preceding strategy: 


Strategy 4 


a) calibrate the structure of the Ist phase sample s, on that 
of the total population U in terms of variable x, , then 


b) calibrate the structure of the 2nd phase sample s simul- 
taneously in terms of variables x, and x, on the basis 
of the weights modified by the preceding calibration, 
that is, 


- on the structure of the total population U as regards 
x| 

- on the structure of s, modified taking account of the 
preceding calibration for x. 


This strategy leads to the following calibration equations: 


Stage a: 


yy ED = > Xi» 


i€Sq ce i¢U 
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which determines 6,, then 


Stage b: 


u 


i€s 


F(x/,B))F (x16, + xi262) 


Tai Mj 


xi = > xX; = X,, and 
igU 


F(x Bi) F (x16) + X262) beage 
DY = 


> 


6 Nagi Tj 
F(x 81) 5 
y Sas ey MIS ADs 
T . 
1€Sq Cs 


which determines 6, and 6). 


When the calibration function is exponential, it is clear 
that strategies 3 and 4 coincide. 

In this calibration-based approach, the viewpoint 
adopted is that of reduction of variance based on the 
characteristics of the sampling plan, without consideration 
of the model. Two questions then naturally arise: 


— Can each of these four strategies be linked to a model- 
based approach? 

- Can these four strategies be compared in terms of 
variance? 


We will first examine the link between the three strategies 
defined by a calibration approach and the strategies 
defined by a model-based or regression approach, after 
which we will focus on calculating the variances of the 
estimators associated with each of the strategies. 


4.2 Link Between the Different Possible Strategies 
and the Regression Approach 


When Fis linear, each of the estimators associated with 
the four strategies may be rewritten simply. 


Notations 


Throughout the rest of this article we will use the 
following notations for a vector of any variable z: 


* 


F(x 61) Zz o* = ys F(x, 81) z 


N» 


i yO 


i€s Tai Ti i€Sq Tai 
We will also omit the 7 indexes in order to lighten the 


presentation when there is no ambiguity. 


Strategy 1 
The weightings are of the form 
F(xi81) ; 
we = —) F(x; 2), 


Nagi Tj 
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the associated estimator Y, may be rewritten by translating 
the effect of the second calibration on x): 


Y,=¥* + [Xs — Xs]'B, with 


, — , 
B, = ee F(x; B;) soxt) (x F(x; B)) wy), 


Tg Tq 
then by translating the effect of the first calibration on x,: 
¥o= ¥ + (X, — X41’ By + (%5 — X5)' B,, 
or: 


Y, = [X{B,) + [X3’B, — X/B)) + (Y- 


with B, = (x aya 


3’ By), 


Y, = (X{ b,) + [XS by — Xf bi) + [LY — X3’ by). 


X,Y 
Tv Grr tle 
oy a a 


Now, Y; is rewritten: 


We thus obtain an estimator similar to the estimator 
Y, that is obtained from the model-based approach in 
cases where the information contained in x, is considered 
to be substitutable for the information contained in x, for 
estimating y and also to be of lesser quality. The differ- 
ences between Y, and Y, concern the following points: 


1. B, is estimated by incorporating the changes from the 
calibration on x;, unlike b5. 


val 
2. The estimate B, = (x =) (x ren) of By, 


TT 
a 
Sq s 


is made in part on s,, unlike },. 


3. Lastly, we use the adjusted weights F(x,6,)/a,7 in 
the sums in x, ons and on S, in Y, in unlike what was 
done for Y;: the estimation on x is improved by the 
knowledge of x. 


Thus the underlying modelling here is indeed: (1) y; = 
Xj,b; + uj and (2) y; = xj2b2. + uj, the second of which 
we think is a priori better for predicting the value of y;. 


Strategy 2 
We obtain weights 
Se F(xj,a, + Xj2Q2) 


i ’ 
Ti Tj 


the associated estimator is rewritten as follows: 


¥; = [X/4,] + [Xa] +4¥ — X/ 4, —X3a)]. 


We thus obtain exactly the estimator Y, proposed in 
the regression model approach in the case in which the 
information contained in x, is considered complementary 
to the information contained in x, for estimating y. The 
underlying model here is indeed y; = xj,)@; + Xj2@) + Uj. 


Strategy 3 
We obtain weights 
Op F(xiayi + X22) 


I 9) 
Tai Vj 


the associated estimator is rewritten as: 


+ [X, — X]' a, + [X$ -— X)]' & 


ot 
ll 
a> 


Yo = (XG,] + [X3'G] + [Y — Xf 4, — X34). 


Now, 


2 s Art \ = Nee 
Ci = QQ + (32 73) (x ) a 
Ta Taq 


We thus obtain an estimator that is close to the esti- 
mator Y; proposed in the regression model approach in 
the case in which the information contained in x, is con- 
sidered complementary to the information contained in 
xX, for estimating y. The underlying model here is y; = 
XjC; + My, (Xi2) ‘C2 + u;. The differences between Y; 
and Y¢ concern the estimated coefficients: (§!) differs 
slightly from (é!) and XG 4, —X3 d,] differs slightly 
from [Y — X;¢, — M,, X3 ¢,]. On the other hand, these 
quantities are asymptotically equivalent. 


Strategy 4 
We obtain weights 


7 _ F(x Bi) F (x16) + X252) 
We 5 


I 


Taji Tj 
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the associated estimator is rewritten as follows: 


A 


Y, = ¥* + [X, — Xt] ae + Ls — Re as. 


By changing the initial weights in d; = F(X; 8,)/tgim; 
we obtain in the same manner: 


Yo = (Xj af] + [X3' a5] + [¥* — Xf at — X3' 43). 


By replacing X% by its expression found above, we 
obtain: 


Y,= [X{ Cf] + (M,, Xs a) + (Y* —X1* at —X3'H), 


with 


Finally, Y, and Y, are asymptotically equivalent. 


Say that w = y — xj df — x34}. Then Y; = Yo + 
[W* — W]. Now, asymptotically [W* — W] is an 
infinitely small negligible before Y¢: 


ti A= (5 =) (DM) le - 4, 


Tepe ‘iby 


and 


wxi\ tends toward $ 1 
— X, —X,| = O| —). 
@ 3) zero and | 1 Ge) 


Ultimately we obtain Y, = Y,. 


In conclusion, when the calibration function is expo- 
nential, the estimator Y7 coincides exactly with the pre- 
ceding. When Fis linear, Y7 is close to the preceding and 
thus still corresponds to the regression model approach in 
the case in which the information contained in x, is con- 
sidered complementary to the information contained 
in x, for estimating y and in which the decomposition 
Yi = XC, + My, (Xj2)' Cy + u; is used. 


Conclusion: The Three Classes of Estimators 


We have just seen that the four strategies derived from 
a calibration approach could be associated with regression 
modelling. We thus obtain three classes of estimators: 


Y, = Yj; associated with the models 


C1) ¥; = x10; + ua, 
and 
Xj2b, + Uj 


(2) y; 
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Y; = Y> associated with the model 


= , t 
Yi = Xj, + Xj2Qq + Ui; 


on 
| 


= Y,and Y; = Y; associated with the model 


Yi = XC, + My, (Xi2) C2 + Uy. 


The approximation =, which indicates that the esti- 
mators are attached to the same regression model, takes 
on its full meaning when we are interested in calculating 
the variance of these different estimators, since the esti- 
mators that are attached to the same regression model have 
the same asymptotic variance. 


5. ESTIMATION OF VARIANCES 


Let us consider the variances of the different estimators 
Y,, ..., Y, defined above. AV designates the asymptotic 
variance of an estimator that is obtained when N, n and 
m tend toward infinity in a constant relationship. 


5.1 Estimator Y, and Y,: model 
Yi = X45, + Uj and (2) y; = X72b. + Up. 


e Estimator Y, 


The variance of this estimator and its estimate are given 
in the work of Sarndal, Swensson and Wretman (1991). 
The variance breaks down into two terms that measure the 
amounts of variance due respectively to the first and the 
second phase of the sampling. 


a yj; Uy; Ud; 
AVY) =()s Ap — ) HE > (een Tea 
el ( ee @ = ). 


i,j€U ifs, Rath Raj 
Veh 2a 
with: Aj = Tj — TT, 
eg er ar 
ih r= Taij Tai Maj ’ 
— , 
Wy; = Yi — x1), 


U2, = Yi — X22, 


= 
by) = ( ye vx) ( dS “ay ’ 
icU icU 
=i 
by = ( ye vax) ( 2 vay). 
i¢U i¢U 


Thus the variance estimator also breaks down into two 
terms that estimate the amounts of variance relating to 
each of the sampling phases. We find that by construction 
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of Y, x; serves to reduce the variance brought about by 
the first phase and x, serves to reduce the variance brought 
about by the second phase. 


V(Y%) = 
IN aT BT et This 
(x 2 ij aaa ath ( x i LLP) ) 
9 
feces TijMaiy TWiT; ie Taij Ti Nai Nj NVaj 
Ist phase 2nd phase 
with: ty; = yi — Xd, 


yj = Yi — Xj2 dp. 


Such a decomposition is based on the expression 
V(Y\) = V(ELY, | s.]) + E(VLY, | 5g]), which will 
apply for all the other estimators. 


e Estimator Y, 


The terms of the development to the first order in 
1/Vm of Y, and Y, coincide exactly. We can therefore give 
a more precise meaning to the expression Y, = Y,. We 
deduce from this that AV(Y,) = AV(Y4). Thus: 


V(¥%,) = 
TR Beri Pe Vann RT By 
ye » ij li ) a ( 5. ij U2; U2; ) 
’ 
pyesines Tij T aij Tj T; jes Taij Tj Nagi Mj aj 
with: Hy; = yi — xX By, 


Ui = Yj — Xj2 By. 


5.2 Estimators Y, = Ys: model 
Yi = XQ + Xa, + U; 


It is easy to show (see Dupont 1994) that: 


AV(Y,) = AV(Y;) = 


Teele 2 UjU; 
(Zatt) + (Eat) 


Tj TM gi Tj Maj 


with: Vp = Yi X},a), 


/ 
3 Xi Xi ys, Xi Xi2 S Xi i 
psa (ain ae i€U icU i€U 
ay , / 
Xj2Xi1 > Xj2Xj2 y Xj2Ji 


i€¢U i¢U ieU 


From this we deduce that: 


V(¥2) = V(¥s) = 


1 mA 2 Ax 
( ye Ay aes ci ( e Ajj uj u; ) 
> 

Tj Naij TjT; i Taij Tj Nagi Nj Maj 


LJj€s 


with: Vi = i xia, 


In this formulation we find that by construction of 
Y, — Ys, x, reduces the variance brought about by the 
first phase and x, and x, are used simultaneously to reduce 
the variance brought about by the second phase. 


5.3. Estimators Y;, Y, and Y;: model 
Yi = Xie, + My, Xr)’ C2 + Uj 
We show that AV(Y¢) = AV(Y7) = AV(Y3). Thus, 
AV(Y;) = AV(Y.) = AViG@Y r= 


Uy j;U4; u;Uu; 
AL ey Vo (ie poms 
(a 3 =) ( is a ), 
» % a 


Tj Nagi Mj Naj 


om / at? vy 
Wy; = Yi — XAC, = Yi — XO, 


— / — , 
Uj; = Vi — Xe, — My, Xj72€2. = Vi — XG, — Xj2Q2- 


From this we deduce the three variance estimators, 
which differ owing to different estimated coefficients: 


V(¥3) = 
Ale aia. A2. iii, 
§ ij Lika A yy U hed] ) 
> 
wee NijMaiy Nit; pigs acd Tj Maj Nj Maj 


A rn 


Uyj = Yi — XC» 


ate a 
Uj; = Yi — XC, — My Xl, 
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We find that by construction of Y3, Ys and Y7, x, is 
used to achieve maximum reduction of the variance 
brought about by the first phase and x serves to reduce 
the variance brought about by the second phase. 


6. CHOICE OF ESTIMATORS WHERE THERE 
IS SELECTION BIAS 


In practice, when a survey is adjusted, it is not unusual 
to want not only to improve the estimation, but also and 
more especially to correct the biases introduced by un- 
controlled selections of individuals, such as nonresponse. 

We shall examine the case of a two-phase sampling 
in which the second phase is equivalent to total non- 
response. The weights z; of the second-phase sampling 
are thus unknown. The calibration of s will enable us to 
estimate these probabilities, while reducing the variance 
(cf. Deville and Dupont 1993). However, asymptotically, 
the corrections of bias to be made to the weights are greater 
than the changes to be made in order to improve the 
estimators. It is therefore the implicit response model that 
will guide the choice between the different estimators: 


The implicit response model for the first class of estimators 
Is pj = 1/F(xj2B)). 


The implicit response model for the second and third 
classes of estimators is p; = 1/F(x/2A2 + Xj,A)). 


e¢ Whatever the response model, an evaluation of the three 
classes of estimators on the basis of the sampling plan 
alone still indicates that the third is preferable, since it 
is appropriate for all the response models. 


e If the strategies are evaluated on the basis of regression 
modelling, we will use the first class of estimators only 
if the response mechanism is well explained by x, that is, 
PD; = 1/F (xB). Now, we have seen that the modelling 
associated with the first class of estimators takes on its 
meaning when the variables x, and x are highly corre- 
lated. It is therefore fairly probable that in this context, 
the variable x, will be sufficient to explain the response 
mechanism. Should this not be the case, it will be neces- 
sary to turn to the third class of estimators. 
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The comparison between the three strategies may thus 
be adapted in a context in which we wish to correct the 
biases introduced by uncontrolled selections. The conclu- 
sions remain largely the same. 

According to the same principle, it is of course possible 
to make comparisons between alternative adjustment 
strategies in the context of samplings that entail more than 
two phases and one or more uncontrolled selections. 


7. A PRIORI AND A POSTERIORI USE OF 
AUXILIARY INFORMATION 


The calibration estimator enables us to improve the 
estimate a posteriori, by reducing the variance and cor- 
recting the bias, as noted above. However, we may want 
to incorporate the auxiliary information a priori, at the 
sampling stage rather than a posteriori at the estimation 
stage. We then encounter, in a more complex context, the 
classical opposition between stratification and 
poststratification, well known in the case of single-phase 
sampling, when all the auxiliary variables are qualitative. 

It is possible to transpose the terms of the choice 
between using the information a priori and a posteriori, 
in the sampling and auxiliary information configuration 
studied, when the auxiliary variables are qualitative. When 
the auxiliary variables are qualitative, a calibration corre- 
sponds exactly to poststratification. 

We saw earlier that in order to determine the proper 
adjustment procedure, it was necessary to distinguish two 
possible modellings of the variable of interest, depending 
on whether the information in x, and the information in 
X» were considered substitutable or complementary. Each 
of these two modellings then led to one or more different 
adjustment procedures. Similarly, these two modellings 
arise when it is a matter of identifying the best sampling 
strategy for incorporating the auxiliary information: 


e¢ When the information in x, and the information in x, 
are substitutable, the modelling of the variable of interest 
is as follows: 


(1) J, = Xj by =P Uj and 


(2) y; = Xj2b2 + uj. where the second model is better 
for predicting the value of y;. 


We have seen that the use of the auxiliary information 
a posteriori leads to calibration strategy No. 1, that is, 
to the first class of estimators. If we wish to take account 
of the auxiliary information at the sampling stage, it is 
natural to propose a sampling stratified on x, for the 
first phase and a sampling stratified on x, for the 
second phase. 


However, the parallel between the adjustment procedure 
and the sampling procedure is not complete: in a cali- 
bration, only the marginal information in x, can be used. 
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This results in incomplete poststratification (Sarndal and 
Deville 1992). On the other hand, in the sampling proce- 
dure proposed as ana priori alternative, we are obliged 
to use all the cross-tabulations of the x, variables. The 
a priori equivalent of a calibration would accordingly 
be a sampling balanced on the margins of the vector of 
variables x, . 


When the information contained in x, and the informa- 
tion contained in x, are complementary, the modelling 
of the variable of interest is y is y; = Xj,b, + Xj2b, + Uj. 
We have seen that in this case the use of a posteriori 
auxiliary information led to calibration strategies 2, 3 
and 4 in estimator classes 2 and 3. If we wish to take 
account of the auxiliary information at the sampling 
stage, it is natural to propose a sampling stratified on 
x1 for the first phase and a sampling stratified on x, 
and x, for the second phase. 


As before, there is no exact parallel between the a priori 
and a posteriori procedures, since the use of the infor- 
mation a priori mobilizes all the cross-tabulations 
between the variables x, and x. 


Thus it is possible to make a choice between incor- 
porating the information either a@ priori or a posteriori, 
and indeed to optimize the sampling plan, when the aux- 
iliary variables are qualitative. The terms of the choice are 
the same as in a single-phase sampling with a single level 
of information. An additional consideration is the multi- 
plicity of strata created by the cross-tabulations of x, and 
X> in the case in which the modelling used is y; = x;,b; + 
X;2b, + u;, which reinforces the advantages of using the 
information a posteriori. 

When the auxiliary variables are quantitative, the choice 
depends on their conversion into qualitative variables, it 
not being possible to generalize correctly except by using 
the parallel between calibration and balanced sampling 
(cf. Deville 1992). 


8. CONCLUSION 


In a two-phase sampling, when two different sets of 
information are available for the total population on the 
one hand and the sample resulting from the first phase on 
the other hand, several strategies are possible when one 
wishes to use the auxiliary information to improve the 
estimation of totals. 

Two different natural approaches have been used to 
derive estimators: a regression model assisted approach, 
which seeks to adapt the idea of the regression estimator; 
and a calibration approach, which attempts to adapt the 
idea of calibration. The estimators obtained by the two 
approaches may be linked together. We generated three 
alternative underlying modellings to which the various 
estimators obtained may be attached. Thus we obtained 


three classes of estimators. Several conceivable calibration 
strategies were eliminated at the outset as irrelevant. 

We have shown that the estimators of a given class, that 
is, the estimators attached to a given model, are asymp- 
totically equivalent; we gave the form of the variances 
derived in the case of a linear calibration function, but with 
asymptotic equivalences, these results remain valid for any 
calibration function. 

For purposes of evaluating strategies, the form of the 
variances indicates, as intuition would suggest, that one 
of the classes of estimators (estimators 3, 6 and 7 (calibra- 
tion strategies 3 and 4)) is preferable to the other from the 
standpoint of variance when the evaluation is based on the 
sampling plan alone. When it is based on a modelling of 
the variable of interest, it suggests that the preferable class 
of estimators is the one associated with the modelling 
adopted. 

In a situation in which the goal is to adjust a survey and 
to simultaneously correct the biases that would arise from 
the use of gross weightings and reduce the variance, the 
conclusions must be adapted. The changes introduced in 
the weighting to correct the biases are greater than the 
corrections to reduce variance. Hence the variables will be 
incorporated into the calibration once it appears that they 
affect the probability of selection and thus participate in 
the creation of bias. 

When the auxiliary variables are qualitative, the choice 
between a priori and a posteriori use of auxiliary infor- 
mation, that is, between using it at the sampling stage or at 
the adjustment stage, still rests on the distinction between 
the two modellings of the variable of interest. 

These results may easily be generalized to the case of 
sampling involving more than two phases. 
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Estimating Some Measures of Income Inequality from Survey Data: 
An Application of the Estimating Equations Approach 


DAVID A. BINDER and MILORAD S. KOVACEVIC! 


ABSTRACT 


We summarize some salient aspects of the theory of estimation functions for finite populations. In particular, we 
discuss the problem of estimation of means and totals and extend this theory to estimating functions. We then apply 
this estimating functions framework to the problem of estimating measures of income inequality. The resulting 
statistics are nonlinear functions of the observations. Some of them depend on the order of observations or quantiles. 
Consequently, the mean squared errors of these estimates are inexpressible by simple formulae and cannot be 
estimated by conventional variance estimation methods. We show that within the estimating function framework 
this problem can be resolved using the Taylor linearization method. Finally, we illustrate the proposed methodology 
using income data from Canadian Survey of Consumer Finance and comparing it to the ‘delete-one-cluster’ 


jackknifing method. 


KEY WORDS: Complex survey design; Gini family coefficient; Lorenz curve ordinate; Low income measure; 


Quantile share. 


1. INTRODUCTION 


The measurement and analysis of economic inequality 
are well covered in econometrics literature from both, 
theoretical and applied aspects, although the theoretical 
issues prevail. Estimation of inequality measures and the 
impact of the design of sample surveys have gotten less 
attention. Variance estimation, unavoidable in statistical 
inference based on these measures, is seldom an issue in 
the relevant econometric literature. It is usually addressed 
under very strong assumptions and under unsustainable 
simplifications of the design or the formulae for the 
approximate variance. In this paper we present a method 
that can handle with ease both the estimation of the 
measures of income inequality and the variance estimation 
of the resulting non-linear statistics. This method is appli- 
cable under a variety of sampling designs. 

In general, a population distribution can be described by 
its cumulative distribution function, F(y) = Pr{Y < y}, 
where Y is the random variable corresponding to selecting 
one population unit at random. Throughout this paper, 
we assume that Y is non-negative. If Y represents income 
then we are interested in the properties of an income 
distribution, such as income concentration, income shares 
for different population shares, low income proportions, 
etc. We are also interested in the quantile function 
(i= Fa (p) = int{y*| Fy), =p). 

The Lorenz curve, for example, depicts the cumulative 
income against the population share. The formal defini- 
tion of the ordinate of the Lorenz curve corresponding to 
the 100 p-th percentile of the population is 


ja vag a (1.1) 


where 


Ep oo 
| aF(y) = p, and | ydF(y) = py. 
0 0 


The finite population form of the expression (1.1), 
more familiar to survey statisticians, is given by 


L(p) = yy Vitis i | De Yi; 
U U 


where U represents a finite population and /{.} is an 
indicator function. 

The income (quantile) share is defined as the percentage 
of total income shared by the population allocated to the 
certain income quantile interval [£,,, &,,], Pi S P2- It is 
equal to the difference of Lorenz curve ordinates 


QO(P1,P2) = L(p2) — L(y). 


In Figure | we give a graph of the Lorenz curve for the 
Weibull distribution with shape parameter a = 1.6, along 
with the 45° axis. For example, one can read from the 
graph that not more than 25% of the total income is 
allocated to the poor half of population, or that the richest 
10% of the population earn 20% of the total available 
income. 


! David A. Binder, Director, Business Survey Methods Division, and Milorad S. Kovacevic, Senior Methodologist, Household Survey Methods 
Division, Statistics Canada, R.H. Coats Building, Tunney’s Pasture, Ottawa, Ontario, Canada, KIA OT6. 
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Figure 1. Lorenz Curve for the Weibull Distribution with Shape 
Parameter a = 1.6. 


The Gini coefficient measures the degree of the ine- 
quality in income distribution. One definition of the Gini 
coefficient is a linear function of the area between the 
Lorenz curve and the 45° axis, normalized to lie between 
0 and 1. The Gini coefficient in Figure 1 is 0.35. The formal 
definition of the Gini coefficient (Nygard and Sandstrom 
1981) is 


1 1 (o-) 
G= te 2 L(p)dp = | [2EO)e—al |ydF(y)- 
0 0 


A more general family of Gini coefficients, given in 
Nygard and Sandstrom (1981) is 


co 


1 
Oh ah ee | J(F(y)|ydF(y), (1.2) 
0 


where J is a bounded and continuous function. For the 
usual Gini coefficient, J(p) = 2p — 1. 

Another measure of income inequality used by some 
economists is the Low Income Measure. This is defined 
as the proportion of the population units whose income 
is less than half the median income for the population. 
Formally, this is 


M/2 
On dF(y), (1.3a) 
0 


where M is the median defined by 


se 1 
dF (y) =e (1.3b) 


0 


For all these measures, we can express the parameter 
of interest, 8, as the solution to the equation 


Juo.oaroy = 0, 


where u (y,@) is the kernel of the estimating equation. This 
estimating equation formulation will be discussed in 
Section 2. In Sections 3, 4, and 5 we give the estimating 
equations for the above measures along with the approx- 
imation of their mean squared error estimates. In Section 6 
we present estimators of these measures based on the 
complex sample design. Section 7 contains an illustration 
based on the Canadian Survey of Consumer Finance data. 


2. USE OF ESTIMATING EQUATIONS FOR 
FINITE POPULATIONS 


The theory for estimating means and totals from finite 
populations is now well established in the statistical litera- 
ture. A formulation which encompasses most estimators 
used in practice is given in Sarndal, Swensson, and 
Wretman (1992). In this section, we briefly review this 
theory and show how it can be applied to more complex 
statistics through the use of estimating equations, as 
described by Binder (1991) and Binder and Patak (1994). 

We begin the exposition of the main idea by reviewing 
the estimation of the population total 7y and the finite 
population distribution function F(y). The estimation of 
the population total is the core of the estimation equations 
approach of Binder (1991) and Binder and Patak (1994). 
Let the population total of the variable Y, be defined as 


a v |vary. 


Note here that F(y) is a step function corresponding to 
the distribution function for the finite population. We 
consider estimators of the form: 


N 
Ty = Y) wis)vi = Y) wils) Yi, (2.1) 


i€és (=i 


where w;(s) is zero whenever the /-th unit is not in the 
sample. Expression (2.1) gives, for example, the Horvitz- 
Thompson (HT) unbiased estimator if 


1/a;, i 
Gs) -| /m;, i€s, 
0, i¢s, 


or the generalized regression estimator if 


wi(s) = Les (Tye = 1) x, / Toa) apices: 
0, i€s, 
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where 7; is the population total of X, and Ty and Ty2 
are the HT estimates of the totals of X and X° variables, 
respectively. 

Similarly, an estimator for the distribution function is 
given by 


NF(y) = )i wi(s){yi = y}, 
i€s 
where 
Lake je SS 


Diy sy} =| 
OMArayt> y: 


We note that F(y) is uniformly and asymptotically design 
consistent for F(y), but it is not necessarily a true distri- 
bution function, unless 


3 wi(s) = N. 
i€s 


In general, and under certain regularity conditions for 
complex designs (Francisco and Fuller 1991), 


F(y) — F(y) —,0, for any y. 


That is, the finite population distribution function, F(y), 
allows a consistent estimator, F(y). This property of the 
F(y) will be used later in proving the consistency of the 
linearized variance estimators for different income statistics. 
Now, we review the application of the estimating equa- 
tions theory to the estimation of any finite population 
parameter ©, that can be expressed as the solution to 


J¥0.04F0) = 0. 


We define the estimating equation estimate for ©, as that 
value of 8 for which 


\20.0)4F) = 0, (2.2) 


where “#(y,@) is an estimate of u(y,0). 


We can rewrite (2.2) as 


0 =|20.0)4F) 
(2.3) 


= [a(y,0) — u(y,0,) ]dF(y) + uo.0,)aFo+ R, 


where 
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R = \140.8) — u(y,0,)] [dF(y) — dF(y)]. 


The decomposition in (2.3) is the basic starting point 
for all the derivations of variance in the paper. For each 
parameter considered we will prove that the remainder 
term, R, is asymptotically negligible. 

Binder (1983) considered the case where #(y,0) = 
u(y,90) and where, for large samples, 


| mo.8) —u(y,0,) ]dF(y) 


dE{u(y,9) } 


= (0 <r 0.) 
Ey le 


+ 0,(|8 — 9,|). 


Note that the remainder term R from the decomposition 
(2.3) should be of order o, ( |8 — 0,]) to be considered 
as asymptotically negligible. 

For most applications u(y,9) does not need to be esti- 
mated by #(y,0). However, for some applications such 
as the Gini coefficient, the function u(y,@) is estimated 
so that formula (2.2) allows for these cases in general. 


Using these approximations, we have 


= 


patents dE{u(y,0)} 
00 6=6, 
* Jx0.04F0) = \w onan, (2.4) 
where 
=i 
dE{u(y,9) } 
‘ ayes Ue We) ,0,)- 
u*(y) We ws u(y,Oo) 


Once we have obtained the expression for u*(y), the 
derivation of the variance of 8 becomes straightforward. 
Since we have approximated 6 — 0, as an estimator of 
a population total of u*(y;)’s, we can use the mean 
squared error calculations for the estimate of total to 
obtain the variance estimate of 0. 


For example, for 0, equal to the ratio, Ty/Ty, we have 


u = y — Ox, 

. 1 

u* = — (y — ox). 
Lx 
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The remainder term in this case is 
R= Jv — Ox — (y — Oox)] [dF(y) - dF(y)]. 


Therefore, 


a= oe [F(v) — F(y) 1x =, 0, 


for any y and any finite x. 


Similarly, for population quantiles, we have 


u TY Oo =p} 


(225) 


* 
| 


1 
— ec Pay 
ean a an Oded?! 


where f(Qp) is the value of the density function at Qo. 
The second expression in (2.5) is an extension of the 
Bahadur representation for sample quantiles, as described 
by Francisco and Fuller (1991). Result (2.5) will be used 
for the ordinates of the Lorenz curve and for the Low 
Income Measure, which are discussed in Sections 4 and 5. 

The remainder term R in this case reduces to R = 
F(6) — F(®,) — F(®) + F(@,). In the case of the 
simple random sample design, Randles (1982) showed that 
R = 0,(n~”). For the complex design situation, under 
some regularity conditions, Shao and Rao (1994) estab- 
lished a similar asymptotic result: first they showed that 
6 = 0;,=s0,(sq, 7), themthatyR = fo,(n- 7), and 
therefore .R = 0, (10. = O;)|). 


3. GINI FAMILY COEFFICIENT 


For the Gini family coefficient, given by (1.2), we 
can use 


u(y,G;) = J[F(y) ly — Gy. 


Binder’s (1983) approach cannot handle the variance 
estimation of the Gini coefficient. For the Gini coefficient, 
rather than deriving the variances by breaking the problem 
into two parts — one for the ratio estimator and the other 
for the variance of the numerator - we use the estimating 
equations approach to solve the problem in one step. 

Ignoring the remainder term in (2.3), we have the 
following approximation: 


0= jutforp — Gyy}dF(y) 


fas J utFon) — J[F(y)] \ydF(y) 

=(G; 4G) [yarn + Juror — Gyy)dF(y). 
Letting 
juifon eV yal Cy) 


od ten — F(y)|J' (FO) lydF(y), 


and 
\Fow LF(y) ydF(y) 


By ~ 
e | | J LF(y) |ydF (x) dF(y) 
0 


= | \ 7 Fo) dF) | dF(y), 


y 


we have that 


Gy Gr |e oaFo», 


J’ (p)F~'(p)dp 


—— 
Hy F(y) 


se DLE (VAY a Cy tee ELEY) I rors]. (3.1) 


For the case of independent and identically distributed 
observations, this yields the same variance result as de- 
scribed by Glasser (1962) and Sendler (1979). To estimate 
the variance, it is necessary to use estimates for py, F(y), 
and G, in the expression for u*. 

We investigate the asymptotic behaviour of the re- 
mainder term R for the usual Gini coefficient G. The 
remainder is 


R = | 2r0F0) — F(y)] — y(G — G)} 
x [dF(y) — dF(y)]. 


Denoting the difference F(y) — F(y) by D(y), the 
remainder can be expressed as a sum of two integrals 
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ee [2»Duv4dd0 - Ke — G)ydD(y). 


The first integral is reduced to zero by the integration 
by parts, so that the remainder is approximated by 


R — (G — G) (jy — py) 


v 


SOG EGON GF) ) Mh US GEA: 
Therefore, we can say that R = 0,(| G — G |). 


4. LORENZ CURVE ORDINATE AND 
QUANTILE SHARE 


The ordinate of the Lorenz curve was defined in (1.1). 


In terms of estimating equations, the following two equa- 
tions are required: 


uj(y,L(p)) = {fy = E,jy — L(p)y, 


Uy) ely, = £5} p. 


The second equation defines the 100 p-th percentile of the 
distribution; whereas the first equation defines the ordinate 
of the Lorenz curve in terms of the 100p-th percentile. 


Ignoring the remainder term in (2.3), we have the following 
approximation: 


0 =| Uy s 61 ~ Ley aFon 
Fa p 
~ | "raFo) - 12) ~ Lee) |yaroy 
Ep 


+ Juv < &,} — L(p)lydF(y). 


The first term of this expression can be further approx- 
imated as 


Sp 
| ydF(y) ee (é, =y Soa sale 


Sp 


and from (2.5) we see that 


1 2 
—& ~ —| — [I{y s &} - pldF(y), (4.1) 
é, — & hee (y =/€,) — pldF(y 


so that 
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A 


(En — En)Epfl(Ep) = - [et =p fp) a wler 


Therefore, to estimate the variance of the ordinate of 
the Lorenz curve, the appropriate linearization is given 
by using 


1 
u*(y) = re acyl sib) 4 pe, — yh (p) |. 
x 


This yields the same result as described by Beach and 
Davidson (1983) for variances and covariances of ordinates 
of the Lorenz curve in the case of independent and identi- 
cally distributed random variables. To estimate the variance 
it is necessary to use é, and L(p) in the expression for 
u*(y). 

To estimate the quantile share O(p;, p») we need three 
equations 


U,(Y,QO(P1,P2)) = Ep, < »y = Ep JY — Q(P1.P2)y, 


UNV) =A E61, 


u(y) = Hy < >} — pr 


Using the same arguments as before, we arrive at 
g 1 
u(y) = Ly — 8p, I ty. = £p5} 
Ky 


a (y = Ey ty = Ep 3 


¥ Deep, — Lisp, ~~) OWi:P2) | 


5. LOW INCOME MEASURE 


The Low Income Measure was defined in (1.3). In terms 
of estimating equations, the following two equations are 
required: 


M 
u,(y,0) Ta ify = 7 2 8, 


iDnty)y STM} =, 


where M denotes the median of the distribution defined 
by the second equation, whereas the first equation defines 
the Low Income Measure in terms of the median. Ignoring 
the remainder term in (2.3), we have the following approx- 
imation: 
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[b= 8) ea 


pei haS M Z 
~ 57 - ms(%) - (6-6) 


[=e 


Using result (4.1) to substitute for M — M, and solving 
for 8 — 0, we obtain 


A 


OBO = Jw oar». 


where 


+ ify < a PO GH) 


The problem with applying this result to estimate the 
variance of the estimated Low Income Measure is that it 
is necessary to estimate f(M) and f(M/2). To accomplish 
this, we could use 


[Ga 


for some suitably small 4. Alternatively, we could perform 
the following calculations, as suggested by Francisco and 
Fuller (1991) for another problem. For a given value of 
&, we estimate the corresponding percentile, 100p. We 
then construct the Woodruff interval for that percentile. 
This is determined by first solving for /, and h, in 


[ty < & —h,} — pldF(y) 


OG 


hy a Ma 
[msef | 0 < §} pid} | 


Juv <&+h)} — pldF(y) 


bt |] SAAS . & 7}. 


hy Ke Y 
J msef | v0» = £} -p\atn} | 


> 


where Z; _ 4/2 is the 100(1 — a/2)-th percentile from the 
standard normal distribution. Then we compute 


Y 
22 ee Jmsef [uty < §} pdr} | 


f(é) = 
(5.2) 


This calculation uses the asymptotic equivalence of £ — £ 
and the estimated sum of the u* (y)’s given by (2.5). 

Wesee that the estimated variance for the Low Income 
Measure may be somewhat complex to compute. The 
estimating functions framework has however provided us 
with the appropriate formulae. 

The discussion about the remainder term in the decom- 
position (2.3) of the low income measure is analogous to 
that made for the case of the quantile estimation (2.5). 


6. ESTIMATION WITH A COMPLEX SURVEY 


Let us assume a stratified multistage design with a large 
number of strata, H, with a few primary sampling units 
(clusters), n,(=2), sampled from each stratum. For 
example, in the Canadian Survey of Consumer Finance 
(SCF) which uses the Labour Force Survey (LFS) vehicle, 
the number of strata is several hundreds and the number 
of clusters per stratum is on average less than six. Let Wy,; 
be the normalized weight attached to the /-th ultimate unit 
in the c-th cluster of the /-th stratum such that the appro- 
priate estimator of mean and the consistent estimator of 
its mean squared error are 


p= ye Whei Yhei 
s 


mse(@) = J) —"— J) (uhe — mh)? 6.1) 


h c 


where Unc = Yi Waci(Ynei — &) and Hy = 1/ny_ Ye Une- 
We use ¥, = Y¥,¥-¥; to denote summation over all 
ultimate units in the sample incorporating all stages 
of sampling. We assume that PSU’s are selected with 
replacement. 

This paper is not concerned with the efficiency of the 
estimators but rather the properties of commonly used 
estimators. An analysis of more complex estimators found 
in the econometric literature is beyond the scope of our 
study. 

An estimator of the finite population distribution 
function is 


Fey) aD Wiel Vier at 
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A consistent estimator of the approximation of the mean 
squared error of the distribution function estimated in y takes 
the form (6.1) where upc = LiWncill (ici < ¥} — F(y))- 

The usual estimate of the finite population quantile is 
the sample quantile 


E, = inf{Ypci€S:F (Yai) = P3 


which is the solution of the estimating equation 


ie Wail l{Ynci = Ep} — p] = 0. 


Accordingly, using result (2.5), the estimator of the 
mean squared error of the p-th quantile has the form 
(6.1) with 


1 
[f(Ep) 


* 
Unc = 


ara ye ee Eph I: 


If the expression (5.2) is used for the estimation of the 
density function /(£), the MSE estimate of the quantile 
£, becomes 


mse, (é,) = Eeay 6.2) 


Z1-a/2 


where D,(£,) = (A; + Ay)/2 = (Ey — &,)/2 is the 
half length of the 100(1 — a)% confidence interval for 
£ p- Inacomplex sample design, h, and h) are obtained as 
solutions of 


A 


See Nyt 
inf {Ypci €S:F (Vaci) = P — Z1—a/2 \mse[F(E,) 1} 


fy =i, +h = 


inf [Yjoj €8:F nei) = P + Z1—a/2 \mse[F(E,)]}. 


The estimator (6.2) was also used by Francisco and 
Fuller (1991). Generally speaking the motivation for (5.2) 
and consequently for (6.2) comes from Woodruff’s (1952) 
confidence interval for individual quantiles. Francisco and 
Fuller (1986) and Rao and Wu (1987) used these intervals 
to derive variance estimators. Although the estimator 
depends on the confidence coefficient, they showed that 
it is asymptotically consistent for any significance level a. 
Rao and Wu (1987) studied the standard errors of quantiles 
for the cluster samples estimated in this manner. Their 
Monte Carlo results suggest that 95% confidence interval 
works well as a basis for extracting the standard error. 
Binder and Patak (1994) obtained a similar form of the 
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variance estimator by using the estimating equations 
approach. 

The estimate of the usual Gini coefficient is the solution 
of the following estimating equation 


Y) Wrci{ L2F (Paci) — 11¥nci — G¥nci} = 0 


Ss 


and takes the form 


PIN 


yy F (Ynci)Vhei = 
s 


where ff = YsWaci Yici- 

The estimate of the MSE of the Gini coefficient can be 
computed using expression (6.1) by replacing u;., origi- 
nally defined by (3.1), with its complex survey form. After 
some algebraic manipulation we obtain the following 
expression: 


. D ipa 
Unc = i ye Whei [4 (Ynci)Ynci + B(Ynci) a (G+ | 
i 
where 
2 Gt+l 
A(y) = F(y) - 5a 
and 


B(y) = > Waci Ynci {Vnci = V}- 


ty 


The Lorenz curve ordinates could be obtained by 
solving a system of estimating equations 


A 


se WreilLYnci = Ep \Vnci — L(P)Ynci) = 0 


Wael Pic E tues la = 40s 


AY 


The resulting estimate is 


L(p) = iD Waci Yhei avis = E,}. 


Ss 


>|l-_— 


To estimate the mean squared error of the Lorenz curve 
ordinates we simply use the values of wu; defined by (6.3) 
in (6.1) 


* _— 
Upc = 


YD Whe nett)! ar =! ep) 


=>|l— 


+ pip — Yai E(p)|. (6.3) 
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Similarly, the mse of the quantile share 


i Waci Vhci ihibers < Vici = el 


Ss 


O(p1,P2) = 


=>|—_— 


is approximated by (6.1) using 
* ] - 3 
Une = oe Wael Vrei = En) LEV hei = Ens} 
pany 
— (Ynei — pL Vnei ¥ Ep,3 


a Pop, — Prép, — Vnei O(p1,P2)]. 


The Low Income Measure defined by (1.3) is estimated 
as 


6 = k(M72) = » Waci 1{Ynci = M/2}. 
KY 


The mean squared error of the low income measure can 
be estimated approximately by the expression (6.1), where, 
(from the equation (5.1)): 


: f(M/2) 
= ea ae ae ci I ich Mi = \/2 
Une sr) du WacilL{Ynci = M} — 1/2] 

a De WrcilL{¥nci = M/2} — 0). 


! 


7. ILLUSTRATION 


The methodology above is illustrated with an applica- 
tion to the family income data collected in the Canadian 
Survey of Consumer Finance (SCF). We use the file on the 
Disposable Income of Economic Families obtained for the 
province of Ontario in 1988. Disposable income is defined 
as total income after tax reported in the survey. The SCF 
uses the framework of the Canadian Labour Force Survey 
which is based on a stratified, multistage design. For more 
details on the sample design see Singh ef al. (1990). 

We estimated the median M, the Gini coefficient G, 
the Low Income Measure 9, Lorenz Curve Ordinates 
and quintile shares Q(0,.2), O(.2,.4), O(.4,.6), O(.6,.8), 
Q(.8,.1.0). Their standard errors are obtained using the 
proposed methodology and the jackknife ‘delete-one- 
cluster’ method. 

We present a brief description of the jackknife ‘delete- 
one-cluster’ method used for this illustration. First, we 
assume that the estimate of the unknown parameter © can 
be expressed as 9 = L(F), where F is the estimated 
distribution function. The estimate of the distribution 
function Fini) obtained from the sample after removing 


the j-th sampled cluster of the A-th stratum (j = 1, ..., 
ny hes ignes eH )ds 


Fei (¥) = YS Anci( 8s) Wneil nei = ¥3 


Ss 


ee} 
> 
| 
as 
fay 
1K 
= 


where Dg eer = Ny = * 5] 


0, h=g,c=yf. 


Then 0 (¢/) = (Fiejy) and the resulting ‘delete-one- 
cluster’ jackknife estimator of the variance of 9 = £(F) 
is 

F tTilcaal © eo es Ps 
var(0) = )) + Se CO aaa o lee 


g=l & j=! 


It is known that the jackknife variance estimator 
performs poorly for quantiles due to its inconsistency 
(Kovar et al. 1988). There are some recent results (Shao 
and Wu 1989, Rao, Wu and Yue 1992) that suggest that 
the ‘delete d’ jackknife and ‘delete-one-cluster’, under 
certain conditions, may have desirable asymptotic prop- 
erties for the variance estimation of non-smooth statistics 
like quantiles or the low income measure. On the other 
hand, for statistics like the Gini coefficient the jack- 
knife estimator of the asymptotic variance is consistent 
(Shao 1993). 

Unlike jackknifing, the estimating equations approach 
is not computationally intensive. It is simple, explicit and 
incorporates the sample design. It provides formulae for 
the asymptotic variance that are easy to program despite 
their complicated form. 

Realizing the limitations imposed by using a single 
sample to make an objective comparison between different 
methods, the purpose of this example is to point out 
differences in the standard errors obtained by the esti- 
mating equations approach and a computationally intensive 
method like the jackknifing. Results are summarized in 
the table below. The direction of the difference in the 
estimated standard errors confirms the overall conser- 
vativeness of the jackknifing method. The difference can 
be attributed to the upward bias of the jackknifing method 
in the case of the median, although the ‘delete-one-cluster’ 
jackknife is preferable to the ‘delete-1’ jackknife. For the 
quantile shares it can be partly explained by the fact that 
upper quantile shares may not cut over all primary sampling 
units but rather perform as separated classes which may 
affect the jackknifing more than the estimating equations 
method. 
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Table 1 
Measures of Income Inequality and Their Standard Errors 


Standard Error 


Measure Estimate Estimating Jackknifing 
Equations ‘Delete-One- 
Approach Cluster’ 
Median 31705 303.3 569.8 
Gini 0.3482 0.005 0.005 
Low Income 
Measure 0.1980 0.00586 0.00613 
Lorenz Curve 
Ordinates 
L(0.2) 0.0561 0.00137 0.00175 
L(0.4) 0.1745 0.00166 0.00194 
L(0.6) O3522) 0.00246 0.00285 
L(0.8) 0.5982 0.00317 0.00393 
Quintile Shares 
QO, 0.2) 0.0561 0.00137 0.00167 
Q(0.2, 0.4) 0.1186 0.00159 0.00221 
Q(0.4, 0.6) 0.1775 0.00157 0.00282 
Q(0.6, 0.8) 0.2461 0.00158 0.00337 
Q(0.8, 1.0) 0.4017 0.00395 0.00451 


8. SUMMARY 


The problem of estimating the variance of complex 
statistics, such as measures of income inequality, have 
eluded statisticians for years. Replication methods such 
as the jackknife are often suggested for estimation. The 
advantage of the linearization approach is that it can be 
used under a wide class of sampling designs and does not 
suffer from the need for intensive computations which 
methods such as the bootstrap entail. Through the method 
of estimating functions and the decomposition given in 
(2.3), we find that some difficult problems can be solved 
more easily. A discussion about the order of the remainder 
term for some of these measures is given as well. A more 
rigorous proof for a complex sample design can be estab- 
lished along the lines given in Shao and Rao (1994). 
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A Reduced-Size Transportation Algorithm for Maximizing 
the Overlap Between Surveys 


LAWRENCE R. ERNST and MICHAEL M. IKEDA! 


ABSTRACT 


When redesigning a sample with a stratified multi-stage design, it is sometimes considered desirable to maximize 
the number of primary sampling units retained in the new sample without altering unconditional selection 
probabilities. For this problem, an optimal solution which uses transportation theory exists for a very general class 
of designs. However, this procedure has never been used in the redesign of any survey (that the authors are aware 
of), in part because even for moderately-sized strata, the resulting transportation problem may be too large to solve 
in practice. In this paper, a modified reduced-size transportation algorithm is presented for maximizing the overlap, 
which substantially reduces the size of the problem. This reduced-size overlap procedure was used in the recent redesign 
of the Survey of Income and Program Participation (SIPP). The performance of the reduced-size algorithm is 
summarized, both for the actual production SIPP overlap and for earlier, artificial simulations of the SIPP overlap. 
Although the procedure is not optimal and theoretically can produce only negligible improvements in expected overlap 
compared to independent selection, in practice it gave substantial improvements in overlap over independent selection 


for SIPP, and generally provided an overlap that is close to optimal. 


KEY WORDS: Linear programming; Sample redesign; Survey of Income and Program Participation. 


1. INTRODUCTION 


The problem of maximizing the expected number of 
primary sampling units (PSUs) retained in sample when 
redesigning a survey with a stratified design for which the 
PSUs are selected with probability proportional to size was 
introduced to the literature by Keyfitz (1951). Typically, 
the motivation for maximizing the overlap of PSUs is to 
reduce additional costs, such as the training of a new inter- 
viewer for a household survey, incurred with each change 
of sample PSU. Procedures for maximizing overlap do not 
alter the unconditional probability of selection for a set 
of PSUs in a new stratum, but conditions its probability 
of selection in such a manner that the probability of a PSU 
being selected in the new sample is generally greater than 
its unconditional probability when the PSU was in the 
initial sample and less otherwise. 

Overlap procedures are applicable when the redesign 
results in either a restratification of the PSUs or a change 
in their selection probabilities. Keyfitz (1951) presented an 
optimal procedure, but only for one-PSU-per-stratum 
designs in the special case when the initial and new strata 
are identical, with only the selection probabilities changing. 
Causey, Cox and Ernst (1985) obtained an optimal solution 
to the overlap problem under very general conditions by 
formulating it as a transportation problem, which is a 
special form of linear programming problem. This proce- 
dure imposes no restrictions on changes in strata defini- 
tions or number of PSUs per stratum. (A similar result had 


been independently obtained by Arthanari and Dodge 
(1981), although they did not discuss the issue of changes 
in strata definitions. Both sets of authors obtained their 
results by generalizing work of Raj (1968).) However, 
there are at least two other difficulties with the procedure 
of Causey, Cox and Ernst which can make it unusable in 
practice, one which is the focus of Ernst (1986), and the 
other the focus of the current paper. 

The first difficulty is that, if the initial sample of PSUs 
was not selected independently from stratum to stratum, 
the information necessary to compute all the joint proba- 
bilities required by this method may not be available in 
practice. An alternative linear programming procedure, 
for use in such cases, was developed by Ernst (1986). The 
Bureau of the Census has used linear programming to 
overlap its demographic surveys on five occasions. On 
four of these occasions (the selection of the 1980s and 
1990s Current Population Survey (CPS) designs, and the 
1980s and 1990s National Crime Victimization Survey 
(NCVS) designs) the procedure in Ernst (1986) was used 
because the initial design was not selected independently 
from stratum to stratum. In particular, as explained in 
Ernst (1986), if the initial sample was itself selected by 
overlapping with a still earlier design then this independ- 
ence assumption generally does not hold, which was the 
key reason why it did not hold for these four redesigns. 

The second difficulty with the optimal procedure is 
that the transportation problem may be too large to solve 
in practice. The Bureau of the Census also used linear 
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programming to overlap the 1990s Survey of Income and 
Program Participation (SIPP) design with the 1980s SIPP 
design, both two-PSUs-per-stratum designs. The initial 
sample for SIPP was selected independently from stratum 
to stratum. However, the transportation problem for the 
optimal procedure would have been too large to practically 
solve for many strata. This is because for each new stratum 
to be overlapped consisting of n PSUs, the number of 
variables in the transportation problem for the optimal 
procedure can be as large as 2” X (5). The largest value 
of n for which a transportation problem with that many 
variables can be solved with the computer facilities that 
we have used is approximately n = 15. 

This paper presents a reduced-size formulation of the 
overlap procedure as a transportation problem which 
decreases the numbers of variables in the SIPP problem 
to (3) canna Vie (3), a striking reduction for mod- 
erate to large values of n. The procedure assumes that the 
initial sample was selected independently from stratum to 
stratum, and hence could not have been used instead of 
the procedure of Ernst (1986) to overlap the CPS and 
NCVS designs. This reduced-size procedure has been 
successfully run for strata with as many as 68 PSUs. In 
contrast, forn = 68, the 2° x (& ) possible number of 
variables for the unreduced formulation is far beyond the 
size of problem that can be solved by any current computer. 
Furthermore, though the reduced-size procedure sacrifices 
optimality in exchange for its size reduction, it does appear 
in practice to yield results fairly close to optimal, as we will 
show. The reduced-size procedure is the procedure that 
was used to overlap SIPP. 

In Section 2 the procedure of Causey, Cox and Ernst 
(1985) is reviewed, to provide background for the presen- 
tation of the reduced-size procedure. 

The reduced-size procedure is presented in Section 3. 
Although the approach has general applicability, for ease 
of presentation it is only described in detail for the case 
when both the initial and new designs are two-PSUs-per- 
stratum without replacement. A small, artificial example 
of the reduced-size procedure is also presented in Section 3. 
This example serves to illustrate the procedure and to 
demonstrate that the ordering of the pairs of PSUs ina 
new design stratum, a key step in the algorithm, affects 
the expected overlap. We also outline in this section some 
analytical results on the comparison between the reduced- 
size procedure and the optimal procedure. Upper bounds 
on the loss in expected overlap from using the reduced-size 
procedure instead of the optimal procedure are stated. It 
is also explained that in certain situations this loss can 
approach two PSUs for two-PSUs-per-stratum designs, 
the worst possible situation. Further details and proofs of 
the results in this section as well as some results in other 
sections are presented in Ernst and Ikeda 1994. 

In Section 4 the performance of the reduced-size pro- 
cedure is presented, both for the actual SIPP production 


overlap and for earlier, artificial simulations of the SIPP 
overlap. The expected overlap for this procedure is compared 
to that for independent selection of the new sample PSUs 
and to an upper bound on the optimal expected overlap. 
The results show that for this application, in contrast with 
some of the theoretical results described in Section 3, the 
expected overlap with the reduced-size procedure is much 
larger than if independent selection had been used to select 
the new sample PSUs, and nearly as large as the optimal 
expected overlap. Also presented are computer running times 
for the reduced-size procedure as a function of stratum size. 


Finally, our conclusions are stated in Section 5. 


2. REVIEW OF THE OVERLAP PROCEDURE OF 
CAUSEY, COX AND ERNST (1985) 


The overlap procedure of Causey, Cox and Ernst (1985), 
like all overlap procedures, conditions the selection of 
sample PSUs in each new stratum in some way on which 
PSUs in the stratum were in the initial sample. This partic- 
ular overlap procedure attains true optimality by making 
complete use of this information and formulating the 
procedure as a transportation problem. We proceed to 
present this procedure. 

First, however, we introduce some notation that will be 
used throughout the paper. Let S denote a stratum in the 
new design. Each such stratum corresponds to a separate 
overlap problem. Let n denote the number of PSUs in S$ 
and let A,, ..., A, denote the PSUs in S. Let J denote 
the random subset of {1, ..., m} such that k € Jif and 
only if A, was in the initial sample, and let N denote the 
corresponding set with respect to the new sample. For 
example, if A, and A; were the PSUs in S that were in the 
initial sample and A, and A; are the PSUs in the new 
Samplessthene/a—a(2eoeand Nes ileoi em lectmri asm 
denote the number of possible values for Jand N, respec- 
tively. Let J;,i = 1, ..., m*, denote the possible values 
for J and let S;, j = 1, ..., n*, denote the possible 
values for N. The goal of all overlap procedures is to 
maximize the expected number of PSUs in N J, while 
preserving the values of the P(S;)’s. 

To illustrate some of these concepts further, consider 
an example for which n = 3. Then n* = 3 if the new 
design is either 1 or 2 PSUs per stratum with the values 
for N, that is the S;’s, consisting of {1},{2},{3} inthe 1 
PSU per stratum case and {1,2},{1,3},{2,3} in the two 
PSUs per stratum case. Suppose PSUs A, and A, were in 
one initial stratum and PSU A; was in another initial 
stratum and there were three PSUs in each of these initial 
strata. If the initial design was 1 PSU per stratum, then 
m* = 6, with the values of /, that is the J;’s, consisting 
of @, {1},{2},{3}, {1,3}, {2,3}; if the initial design was 
2 PSUs per stratum then m* = 6, with the J;’s consisting 
Of*f1} 9623 SOD afl B i233 {8} 
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We now present the transportation problem for the 
overlap procedure of Causey, Cox and Ernst (1985). 
Abbreviate by P(J;) the probability that 7 = J; and by 
P(S;) the probability that N = S;. In addition, let x,; be 
the variable denoting the joint probability of these two 
events, and let cj; denote the number of elements in 
J; 1 S;. The P(J;)’s, P(S;)’s and c;;’s are known values, 
while the x;;’s are variables for which the optimal values 
are to be determined. Then the transportation problem to 
solve is to determine x;; = 0 which maximize 


m* 


ye Ly Cij Xij (al) 


i=1 j=l 


subject to 


a 
T 
v 

x 
T 


., m*, (2.2) 


KP (Sy) paty col gles eon ™ (2.3) 


Note that in this transportation problem, the objective 
function (2.1) is the expected number of PSUs in S that 
arein N f/ J. Also note that the constraints (2.2) and (2.3) 
are required by the definitions of the P(J;)’s, P(S;)’s 
and the x;;’s. 

Once the optimal x;;’s have been obtained, the condi- 
tional probability that N = S; given that J = J; is then 
xj;/P(J;) for all i,j. 

We present an example to illustrate the use of the for- 
mulation (2.1)-(2.3) in the case where both the initial and 
new designs are two-PSUs-per-stratum without replace- 
ment. In this example, and throughout the paper, p;, 7; 
denote the predetermined probability that i € Jandi € N, 


respectively. 
Consider a final stratum S with n = 3. All of the PSUs 
were in different initial strata. Let pj = .6, py = .75, 


Pea ay = 2), 1>(— .8, Ws = «7. since the PSUs 
were all in different initial strata, there are 8 different 
possibilities for 7, with probabilities given in Table 1. 


Table 1 
Probabilities for Possible Sets of Initial Sample PSUs 


i 1 2 3 4 5 6 i 8 


Jj {1,2,3) (1,2) (1,3) (2,3) (1) (2) 3) O 
Peels lo) OS 2] 045.09" 07.03 


Since the new design is two-PSUs-per-stratum without 
replacement, there are 3 different possibilities for N, 
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namely the/pairsS,5=—! (1279 Se" (13 MS =" {233}, 
andihencet? (S71) 30,25 s) P= 5 20): PS; 50) 

Furthermore, the values of c;; are then as given in 
Table 2. Upon maximizing (2.1) subject to (2.2) and (2.3) 
with the given P(J;)’s, P(S;)’s and c;;’s, an optimal set of 
x;;’8, presented in Table 2, is obtained. Finally, by dividing 
each of the x;; entries in row i of Table 2 by P(J;), an 
optimal set of conditional probabilities P(S; | J;), is 
obtained. For example, since x. = .025 and P(J,) = .315, 
it follows that P(S, | J,;) = 5/63. 


Table 2 


Values of c;; and Values of x;; that Maximize Overlap 
for Optimal Procedure 


Cij xij 

J J 
I | 2 3 1 % 3 
1 2, 2 2 .000 .025 290 
2 2 1 1 BS .000 000 
3 1 7) 1 .000 .105 000 
4 1 1 ”, .000 .000 210 
3) 1 1 0 .045 .000 000 
6 1 0 1 .090 .000 000 
7 0 1 1 -000 .070 000 
8 0 0 0 .030 -000 000 


For this example, as can be computed from (2.1) and 
Table 2, the expected overlap under the optimal procedure 
is 1.735 PSUs. In comparison, the expected overlap if the 
initial and final designs are selected independently is 
Dim, + Pot. + P3773 = 1.39 PSUs. 

For two-PSU-per-stratum without replacement problems, 
the possible values for N are always the (3) subsets of 
Gl eee nol size that is n= (3). However m* can 
vary widely. m* = (3) when the PSUs in S comprise a 
single initial stratum. The upper bound of 2” on m* is 
attained when all the PSUs in S were in different initial 
strata, as illustrated by the previous example, and in some 
other situations. A general, exact expression for m™* is 
presented in Ernst and Ikeda (1994). 

For the two-PSUs-per-stratum without replacement 
overlap problem, the number of variables in the transpor- 
tation problem for the optimal procedure is m*n* which 
can be as large as 2s(3,). Forn = 15, 2"(3) = 3,440,640, 
which is about as large a transportation problem as can 
be solved with the computer facilities that we used. How- 
ever, > 15 for nearly half the nonselfrepresenting strata 
(that is strata consisting of noncertainty PSUs) in our SIPP 
application, and consequently it was necessary to develop 
a procedure, described in the next section, which reduces 
the size of the transportation problem, while still producing 
nearly maximal expected overlap in practice. 
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3. THE ALGORITHM FOR THE REDUCED-SIZE 
PROCEDURE 


Previous work on reducing the size of the transportation 
problem (2.1)-(2.3) has focused on accomplishing the size 
reduction while retaining optimality. For example, the 
approach of Aragon and Pathak (1990) retains optimality 
and reduces the size of the problem by 75 percent when 
m* = n*, Unfortunately, when m* is much larger than 
n*, which is when size reduction is most needed, their 
method produces negligible size reduction in relative 
terms. A generalization of this approach is presented in 
Pathak and Fahimi (1992), but there is no indication that 
their procedure always yields a size reduction that is 
substantial in relative terms. 

In this section a reduced-size procedure is presented 
which takes a different approach. We sacrifice optimality, 
at least in theory, in return for an assured size reduction 
down to a manageable size transportation problem. This 
size reduction is accomplished, in the case when the initial 
and new designs are both two PSUs per stratum for 
example, by ordering all pairs of PSUs in a new stratum 
and then conditioning the new selection probabilities for 
any initial set of sample PSUs of size greater than 2 on the 
first pair of PSUs in the ordering contained in the initial 
set, rather than conditioning on the entire initial set. That 
is, each possible initial set of sample PSUs which consists 
of more than 2 PSUs is combined with a set of size 2. As 
illustrated in Section 4, this procedure may yield a near 
optimal overlap in practice; particularly with an appro- 
priate ordering of the pairs of PSUs, as described in 
Sectionasnle2: 

The reduced-size procedure is applicable whenever 
PSUs in the initial and new designs are selected without 
replacement. However, the procedure will be described in 
detail, in Section 3.1, only for the case when both the initial 
and new designs are two-PSUs-per-stratum. Then, in 
Section 3.2, the changes necessary to apply this procedure 
for other initial and new designs will be sketched. Finally, 
in Section 3.3, some analytical results are outlined on the 
relationships among the expected overlap for the reduced- 
size procedure, the optimal procedure and independent 
selection. It is assumed throughout this section that PSUs 
in the initial sample were selected independently from 
stratum to stratum. 


3.1 Reduced-Size Procedure When Both Designs Are 
Two-PSUs-Per-Stratum 


The reduced-size procedure to be described includes the 
following key aspects: the specific ordering of the pairs of 
PSUs; the reformulation of the transportation problem 
(2.1)-(2.3) for the reduced size procedure; the computation 
of the probabilities for the initial outcomes for this formu- 
lation; and the computation of the cost coefficients (the 


cs) in the objective function. In Section 3.1.1 we present 
a detailed outline of the reduced-size procedure, including 
the reformulated transportation problem. The ordering of 
the pairs is described in Section 3.1.2. Finally, the compu- 
tation of the probabilities for the initial outcomes and the 
cost coefficients are given in Section 3.1.3. 


3.1.1 General Outline of the Procedure 


The general outline of the procedure is as follows. First, 
the (3) subsets of {1, ..., 1} of size 2 are ordered ina 
manner to be described later. (For now, we simply note 
that any ordering can be used to reduce the size of the 
transportation problem. The specific one used is for the 
purpose of accomplishing the size reduction while also 
attempting to give up as little as possible of the gains in 
overlap that the optimal procedure yields.) We let J;, 
js — Teen (3), denote the i-th element in the ordering; 
let ue ts Sees 1) +n be the 7 singleton subsets; and set 
sa +nt1 = ©: Thus, the J;’s constitute all subsets of 
{l, °..,} of 2 or fewer elements. For each possibility for 
I, aunique set /* is associated among these (3) + n + 1 
subsets and the new selection probabilities conditioned on 
the associated /*, rather than on / itself. Therefore, the 
new selection probabilities are conditioned on (3) +n + 1 
events instead of a possible 2” events, which is the reason 
for the size reduction. The associated J* is the first J; for 
which J; C J. That is, if J consists of at least two integers, 
the associated J* is the first pair in the ordering contained 
in J, while if J is a singleton set or empty then /* = J. 

The reduced-size transportation problem attempts to 
retain the PSUs corresponding to elements in the asso- 
ciated set 7* in the new sample, but does not use infor- 
mation on elements in/J ~ J/*. The form of this reduced- 
sized transportation problem based on the set of J;’s is as 
follows. Let p* be the probability that 7* = J;, i = 1, 

a (3) + n+ 1, and abbreviate 7* = P(S;), 
e=PT, Ae § (4). For each i,/, the variable x;; is the joint 
probability that /* = J;and that N = S;, while c;; is the 
expected number of elements in J S; given J* = Jj. 
The problem to solve is to determine x;; = 0 that maximize 


> > Citas (3.1) 


subject to 


(3) 
\> xy = PF, f=ten(Q) tne, (3.2) 
j=l 


(3) +n41 
a * 
de a pas 


: n 
~ im, pes lL; ems 3) (3.3) 
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Once the optimal x;;’s have been obtained, then the con- 
ditional new selection probabilities for S;,j = 1,..., (3), 
given /* = J;, are x;;/p*. Note that the number of variables, 
xj, in the formulation (3.1)-@.3) is (3) +. + 1) x (3), 
in comparison with a maximum of 2” x (3) in the formu- 
lation (2.1)-(2.3). 

It remains to explain the general method for obtaining 
the ordering of the (3) pairs and the procedures for com- 
puting the pj*’s and c;;’s. Before doing this, we present an 
example of the reduced-size procedure, namely the two- 
PSUs-per-stratum example used in Section 2 to illustrate 
the transportation problem formulation for the optimal 
procedure. 

The ordering of the pairs for this example, as will be 
shown later, is {2,3}, {1,2}, {1,3}. Consequently, the 
I;’s, are as given in Table 3. Note that if J = {1,2,3} or 
I = {2,3}, then the associated set is J, = {2,3}. For the 
other six possibilities for J the associated set is J itself. 


Consequently, from Table | we obtain that 


py = PU = (1,2,3)) + PU = (2,3}) =.525, (3.4) 


Pr UP LIke Sy and pr PC; si) itary OST, 
yielding the values in Table 3. Since uF = P(S;), we 
havea? 430, «3 = 220, af -='.50. 


Table 3 
Probabilities of Associated Sets: Reduced-Size Procedure 
I 
1 2 3 4 5 6 7 


dip 4253} orb ls2 We! M633 rif 2 We BES 
FSS 5S gk 135-1) 105 > 14045 1097 \ sn 07# 4203 


The cj values for this example are given in Table 4. 
In order to obtain these values, we simplified the compu- 
tation by letting 


Dit = P(t.e 7 | J* = Ib 
‘ n 
— bel Gh Piao lela alawc este (3e5) 


and noting that if S; = {s,¢} then 
Cij = bis = De. (3.6) 


That is, the expected number of elements in J M S; 
given /* = J; is simply the sum of the probabilities that 
each of the two elements in S; was in J given J* = J;. Also 
observe that while the transportation problem for the 
optimal procedure knows the exact value for J and hence 
knows with certainty whether each element in S; was in J, 
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this is not the case for the reduced-size procedure, since 
only the associated set J; is known. To illustrate, consider 
the first row of Table 4. Since J; = {2,3}, we know that 
2 € Tand3 € J, and hence by = 5,3 = 1. However, we 
do not with certainty whether 1 € J since J; is the asso- 
Ciatedisetfor both = (1-253) andw = {253}. Infact, 
from Table 1, 


7 PU = {1,2,3}) $8 
PPE 193 y)eeera aay 


bi 


Then cy; = 5i, + By. = 1.6, with cj, c;3 computed 
similarly. For the remaining six rows in Table 4, J; = J 
and hence it is known with certainty which integers were 
in J. Consequently, the c;;’s for these six rows are easily 
computed. 

Finally, we maximize the expected overlap (3.1) subject 
to (3.2) and (3.3), obtaining the x; values in Table 4. The 
conditional probabilities P(N = S; | /* = J;) in Table 5 
are then obtained by dividing each of the x;; entries in the 
i-th row of Table 4 by p*. 


Table 4 


Values of cj; and Values of x;; that Maximize Overlap 
for the Reduced-Size Procedure 


Cij xij 

J id 
i I; 1 2 3 1 2 3 
Tega 3 ln 160.04. 0/000. 0025... 0500 
met Oe? Ome 1 Oa al Of 0.1355. 0,000 2,0:000 
Semen (il 3} uhelO tae? OGnel 0» 60.000jg20. 105 a9. 0.000 
4 {1} 1 OSPR OTTS0. 05% 4.05045), 1140:000 $9 0/000 
5 {2} 10 0.0 1.0 0.090 0.000 0.000 
6 {3} 0.0 1.0 1.0 0.000 0.070 0.000 
7 7) 0.0 4,0.0. 50:0. 0.030. = 10.000..= 0:000 


Table 5 
Conditional Probabilities for the Reduced-Size Procedure 


u 
i If. 1 2 3 
1 {2,3} 0 1/21 20/21 
2, (152) 1 0 0 
3 {1,3} 0 1 0 
4 {1} 1 0 0 
5 {2} 1 0 0 
6 {3} 0 1 0 
7 DO 1 0 0 
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The expected overlap for the reduced-size procedure is .01 
less than optimal, that is 1.725 PSUs. The deviation from 
optimality arises solely because the expected overlap is 1.6 
for the joint event that J* = {2,3} and N = {1,3}. Since 
the probability of this joint event is .025, and the optimal 
procedure for this example always produces an overlap of 
2 when at least 2 of the PSUs were in the initial sample, 
the deviation from optimality is .025(2 — 1.6) = .01. 

The reason that the reduced-size procedure is not able 
to obtain optimality is that the pair {2,3} has a smaller 
probability of selection in the new sample than in the initial 
sample. As a result, both the optimal procedure and the 
reduced-size procedure must sometimes select another pair 
(always {1,3} for both procedures in this example) when 
{2,3} was in the initial sample. The distinction between 
the two procedures is that the optimal procedure only 
selects {1,3} when 1 € J. The reduced-size procedure is 
unable to use the information about whether 1 € J. Asa 
result, when {2,3} C J, 1 € Nindependently of whether 
1 € J. This results in a deviation from the optimal overlap. 


3.1.2 The Ordering of the Pairs 


We now proceed to show in general how the ordering 
of the pairs is obtained. We use the additional notation 
heresthatip paw Ssa hi a leas Ser set bis ne, OME 
probability that s, t € Jands, ¢ € N, respectively. 

The motivation for the ordering of the pairs is as 
follows. If the i-th pair in the ordering is {s,¢} then it would 
be possible for the transportation problem to retain this 
pair in the new sample when J* = J; with conditional 
probability min{1,z,,/p*}. (The conditional retention 
probability cannot be any higher than this, since a higher 
value would result in an unconditional selection proba- 
bility for the pair in the new design exceeding z,,.) There- 
fore, roughly the goal in the ordering is to make these 
conditional probabilities as large as possible on average 
over all pairs. 

To illustrate how the ordering of the pairs affects the 
expected overlap we consider the example of Table 3. Our 
ordering procedure, as will be shown later, produces the 
indicated ordering and yields an expected overlap of 1.725 
PSUs. Next consider the following alternative ordering for 
this example. Let the first pair in the ordering be {1,3}, the 
second pair be {1,2} and the last pair be {2,3}. With this 
alternative ordering, 7* = {1,3} whenever either J = 
{1,2,3} or J = {1,3}. Therefore, for this ordering p} is 
the probability that * = {1,3}, whichis now .42. Further- 
more, for this alternative ordering, pf = P(J* = {2,3}) = 
PC = {2,3}) = .21, while the other 5 columns in Table 3 
remain unchanged. The alternative ordering results in a 
table of conditional probabilities similar to Table 5, except 
that in row | the J;, 7 = 2 andj = 3 columns now become 
{1,3}, 10/21 and 11/21, respectively, and in row 3 the 
corresponding columns are now {2,3}, 0 and 1, respectively. 


It can be calculated, using the same approach used for 
the original ordering that the expected overlap for the 
alternative ordering is 0.055 less than optimal, that is 1.68 
PSUs. The reason that this alternative ordering results in 
a lower expected overlap is as follows. In general a later 
placement of a pair in the ordering, results in a lower value 
for the corresponding p*, and hence a higher conditional 
retention probability when /* = J;. That is, with {1,3} 
first in the ordering, 713/pf = 10/21, which is the condi- 
tional retention probability for this pair when /* = {1,3}; 
while when {1,3} is third in the ordering, 7,3/p¥ > 1 
and this pair is retained with certainty. Now the conditional 
retention probability for the pair {2,3} when /* = {2,3} 
also increases to 1 when {2,3} is moved from first to third 
in the ordering, but the increase is only from 20/21, and 
hence the original ordering in Table 3 produces a higher 
expected overlap than the alternative ordering. 

Thus, as this example illustrates, the goal of the ordering 
is to place pairs earlier in the ordering that have a relatively 
high conditional retention probability even with an early 
placement. To obtain the desired ordering of the pairs of 
INtesers,sanionderingsfiGl)) meee C72) L Ole (alemerar Tt mwa 
first be obtained by recursion. Then corresponding to each 
k= 1, 4%.,.2.— 1, an ordering’g¢(1) 5. 28 ee) 
Ofsli tee) = iG) eee ACs) e) will. beiconstmicted 
by recursion. A linear ordering of the distinct pairs in 
{1,...,m} would then be determined as follows. Each 
such pair can be represented uniquely as an ordered 
pair (Ch )y ec) metorssom erknce lees 
Qe {1,...,n — k}. A second pair representable in the 
form (f(k’), g,(£’)) precedes (f(k), g,(£)) if and only 
if either k’ < k, ork’ =kand?’ < #. To illustrate, for 
the example just considered it will be shown later that 
SC) = 2, f(2) = 3, f(3) = 1, gC) = 3, 2) (2) = 1, 
g>(1) = 1, and hence the ordering of the pairs is {2,3}, 
{2,1}, {3,1}. Both the f ordering and the g, ordering will 
be constructed to meet the goal stated at the beginning of 
this paragraph. 

To obtain the ordering f(1), . 
Genin PU), Me = ly oc 
satisfying 


..,f(n), recursively 
.» n, by choosing f(k) € T, 


Tf (k) Diy = max{ 7;/p;*: 1é T,}, 


where 
Witanthie.., nh. Tes Ty = aS Che 
ie) Ps 1, 0D) aPC al eat eae ane 
Kya 1 hae ee (Gen) 
Since p{!) = p,, the ordering just defined corresponds 


to placing first a PSU with the greatest value of 1;/p*. 
For all k, Pie is the probability that f(k) was in J and 
none of the k — 1 elements preceeding f(x) in the f 
ordering were in J, and hence Pie) is the probability that 
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an attempt is made to retain A;,,) in the new sample 
either as the first member of an ordered pair of initial 
sample PSUs or as the only initial sample PSU in S. Gener- 
ally, the larger my(,) liga is, the greater the probability 
that this attempt would be successful. Thus, the moti- 
vation for the f ordering of the individual PSUs is the 
analog of the motivation for the ordering of the pairs of 
PSUs that we previously discussed. 

It remains to explain how to compute p{*? for k = 2. 
To this end, let r denote the number of initial strata with 
PSUs in common with S and let F,, a = 1, ..., 7, denote 
a partition of {1,...,} such that /and/ are in the same 
F,, if and only if A; and A; were in the same initial stratum. 
Then let 


Pot ell OVP CuL) TT, 


RE Kia VEY PG SES) 


pi(T) =P(i€l and INF,CT), a=1,...,r, 


LIGA OES eH) Tee, 543,29) 


and observe that 


PAE SP e Dock HE trpgeeine G10) 
i€hy~T i, j€Foy~ T 
i<j 

Pin(T) =pi- Y) Dy; (3.11) 


j€Fy~T 


and finally, as established in Ernst and Ikeda (1994), 


PM = picT) T] Pi(Te), = Lean, 


f=} 
(Aa 


Henin en .G:t2) 


Next, for each k = 1, ..., m — 1, the ordering g,(£), 
@€=1,...,n — k, is recursively defined by choosing 
gex(l) € Ty, satisfying 

/p‘) = max{ / () Sea 
TS (Kk), BK(O)/P fk), 8K) TW (k),j/P f(b), 5° I kes» 
where 
Ty = i Ieee | ow CL) BSH HOS EI 
Deel kict), = USEME ye eat 
The = Tre U {(f(k)}, 
Phi = PUK), je T and P'CTR), 


a ee a ee OES 
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Note that p/?,, ; is thus the joint probability that f(x) 
is the first integer in the f ordering in J, that none of the 
first 2 — 1 integers in the g, ordering are in /, and that 
J € I. Consequently, Pree), ell is the probability that 
IF (Kk) se, (f)} Furthermore, if J, =" (fk); 27(1)} 
thenip*i= Prtk),a1A0» and hence the choice of g,() results 
in the largest value of my(x),.,(/pi among the elements 
in 7,,.in accordance with the previously stated goal for the 
ordering of the pairs of PSUs. 

To compute ay j> it is established in Ernst and Ikeda 
(1994) that if f(k) € F,/ € Fp, then 


a 
PRley7 = Pres [] oi (TR) if w= 8, 


t=1 
(Aa 


(3.14) 


= Pf k),o( TH) Pip (The) II Di (TR) if a ZB. 


t=] 
t4#a,B 


We illustrate the computations used in obtaining the 
ordering for the example that we have been considering. 
First note that f(1) = 2 since the largest value of 2;/p; 
occurs for i = 2. Next we find g,(1) which, since 
f(1) = 2, is the / € {1,3} with the maximum value of 
;/p3}). To find this j, first let F, = {a}, oe = 1,2,3, 
and note that 7#; = {1,2,3}. From (3.14) witha = 2, 
@ = 1, it then follows that 


PSD = pH12,3} pi (12,335 (1.2.3) = pop - 1 = .45, 


and similarly it can be obtained that p{}) = .525. Hence 
ert he=—w9, Since..5/ 2525 "237.45; Therefore; the: first 
pair in the ordering is {f(1), g,(1)} = {2,3}. Then 
g,;(2) = 1, since | is the only integer remaining to be used 
in the g, ordering, and consequently the second pair in 
the ordering is {f(1), g,;(2)} = {2,1}. It is not really 
necessary to determine f(2), since {1,3} is the only 
remaining pair, and hence the last pair, but to further 
illustrate the computations, observe that 7, = {1,3}, 
pi = pii(1,33p3 1,335 (1,3) =p1(1 — pa) +1 = 15 
by (3.12), and similarly p{?) = p;(1 — p)) - 1 = .175. 
Bience f(2)e—s3ysince.7/2175.2> 25/-15sConsequently, 
g(1) = 1, f(3) = 1. 


3.1.3 Computation of pj and c; 


Next we explain the computation of the p*’s. If J; 
consists of the pair of integers J; = {f(k), g,(0)} then, 
as previously noted, p* = PHd g4l0 Consequently, p* 
can be computed from (3.14) withy = g;,(?). 

If J; is a singleton set {t} for some ¢ € F,, then, as 
established in Ernst and Ikeda (1994), 
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pF = paltt}) T] pul@). (3.15) 


u=1 
uUuFAa 


Finally, if 7, = @, then 


pr = []| pu (©). 
u=1 


It remains only to explain how to compute the c;;’s 
which, by (3.5) and (3.6), reduces to computing bD;,, 
fi Loa ctery. (Coe nasteeti aren Le Larne ee are 

To compute b;,, observe that 


=) ite ay ean uaa Vs 
=O) tf i ty) sealer al), 


while if J) = {f(k), g,(0)} and f(k) € Fy, ex(f) € Fe, 
i¢ F.tben 


b, =1 if t=f(k) or t= 2;(2), (3.16) 


On if ot ¢. To, (3.17) 


O Nile the, Taal en()) 
Buk a — eve, se) 


Dp 3 
= Mt if 6 € Tre ~ (a(2)} 
PF (k) (TR) 
and: ye=704~~6,216.19) 


7 fe — Merce creep 
Pz,(0),6( TK) 
and "y= "6" a; "G.20) 


_ Pw (Th) ‘peer: 
Mois. ~ {gx(t)} 
D5 (TK) ‘s 3 


ANGRY eH FOnay as Bua SeaL) 


In Ernst and Ikeda (1994) it is demonstrated how 
(3.16)-(3.21) were obtained. 

In the actual implementation for the SIPP application, 
modifications of the reduced-size procedure were needed 
to overlap the 1990s SIPP design with the 1980s SIPP 
design. The modifications were necessary because the PSU 
definitions in the 1980s and 1990s designs were not iden- 
tical. As a result, some PSUs in the 1990s design could 
intersect more than one 1980s design PSU. These modifi- 
cations are detailed in Ernst and Ikeda (1994). 


3.2 Modifications of Reduced-Sized Procedure for 
Other Designs 


In general, consider any m’-PSUs-per-stratum without 
replacement initial design and any m-PSUs-per-stratum 
without replacement final design, where m’, m are any 
positive integers. Although the reduced-size procedure in 
Section 3.1 was only presented for the casem = m’ = 2, 
it is actually applicable for any m, m’. We will sketch the 
modifications necessary when m # 2 or m’ # 2. 

A different value of m’ only requires modification of 
some of the computations. For example, if m = 2, but 
m’' # 2, then the computations for p{*, Brees and ¢; 
would be different but their definitions would not change. 

If m = 3, then, regardless of the value of m’, the set 
of all distinct triples, instead of pairs, of integers in 
(Ree 77 lS Ordered wal tevmconsisisnolmateleastmrnnce 
integers, then the new selection probabilities are conditioned 
only on the first listed triple in the ordering contained in 
J. Otherwise, the new selection probabilities are conditioned 
on / itself. Thus the new selection probabilities are condi- 
tioned on (3) + (3) + 1 + 1events. 

To obtain the desired ordering of the triples of inte- 
gers, first the orderings f(1), ..., /(n)ande,()) jaar 
g,(n — k) are constructed exactly as in the case m = 2. 
then scorrespondine:toleachice—s lea ere — 
...,4—k — 1, an ordering hy), ..., Ag(n —-k —2& 
OL esa It) ae PCL) je tt Ghd CEG) cee nC netS 
constructed in a manner similar to the construction of 
g.(1), ...,e@,(" — k). For example, in defining h,,(v) 
for v = 2, pf}, ; in the definition of g,(£) is replaced by 


P(F(K), &x(),7€ 1 and IC (Th U g()) ~ 


{Age(1), ..., Age(v — WEN 


A linear ordering of the distinct triples in {1, ..., 7} 
is then determined by representing each triple uniquely as 
an ordered triple of the form (f(k), g,(0), Ap(v)). A 
second triple (f(k’), g,(0’), Aye (v’)) precedes the first 
if and only if either k’ < k, ork’ = kand?’ < £@, or 
KC = Ke ayia” = Perel py? < w, 

For m = 4, ordered m-tuples would be defined in a 
similar manner and the new selection probabilities condi- 
tioned on (jm) + (m2 4)... + + Levents. 

For m = 1, the new selection probabilities are conditioned 
on the first member of the ordering f(1), ..., f(”) in 
Iidl 2 QO, OF OC Dnt il = @. 

Note that if m > m’, it is possible that at least some 
ordered m-tuples cannot be subsets of J, in which case 
all such subsets should be excluded from the ordering 
and the set of events on which the new selection proba- 
bilities are conditioned. If no m-tuple can be a subset of 
I, then the new selection probabilities are conditioned on 
I itself. 
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It is not necessary to limit the initial events used in the 
transportation problem to subsets of J of size m or less. For 
example, if m = 2 and (3) ae (3) + n+ lis sufficiently 
small, then a procedure conditioned on subsets of three 
or less can be used, resulting in a generally higher expected 
overlap. Conversely, if (jm) + (m1)... +24 Lis 
too large, the new selection probabilities can be condi- 
tioned on subsets of Jof sizem” or less, wherem” < m, 


although with a generally smaller expected overlap. 


3.3. Relationship Between Expected Overlap for the 
Reduced-Size Procedure, the Optimal Procedure 
and Independent Selection 


Let 0;, Qe, Qo denote the expected overlap for the 
independent selection, the reduced-size procedure, and the 
optimal procedure, respectively. In Ernst and Ikeda (1994) 
the relationship between these quantities is explored. We 
briefly summarize here some of the results. 

It is established that Q; < Qr < QoQ for any m, m’ 
where m, m’ are as in Section 3.2. In addition, for the 
case that we have been focusing on, m = m’ = 2, lower 
bounds are established on Qpg and upper bounds are 
established on Q9 and Qo — Qp. 

For example, let , denote the probability that there 
are at least two elements in J, », denote the probability 
that J is a singleton set, and 


=hnin{min{my/peid = 14.0097} 
TUT aM P NT el OR BRT YA AE 


Then Q9 S 2u. + wy, Qe = AQ2pu2 + pw, /2), and 
Qo — Qe = 2(1 — Ny. + C1 — VZ2)p)- 

Unfortunately these bounds are not always very tight. 
However, in certain circumstances they are useful. For 
example, if 1;; = pj for all i,j and the probability is 1 
that there is at least two elements in J, then it follows from 
these bounds that Qe = Q9 = 2. 

Finally, an example is presented to illustrate a worst 
case situation for Qp in relation to Qo for the case m, 
m' = 2. It shows that Qg may equal 2, while Qp is arbi- 
trarily close to 0. Thus, at least in theory, the reduced-size 
procedure can be ineffective. However, in practice, as will 
be shown in the next section, Qp is much closer to No than 
to Q;, at least for the SIPP application. 


4. APPLICATION OF REDUCED-SIZE 
PROCEDURE TO SIPP 


Results from simulations of the SIPP overlap, done 
prior to production for research and testing purposes, are 
presented, as well as results from the actual SIPP produc- 
tion overlap. Further details are given in Ernst and Ikeda 
(1992b, 1994). 
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In the implementation of the reduced-size overlap 
procedure, minimum cost flow (MCF) optimization soft- 
ware, written by Darwin Kingman and John Mote at the 
University of Texas at Austin, was used to solve the 
required transportation problem. A FORTRAN program 
was written to produce input to and process output from 
the MCF software. 

To test the software prior to production, the program 
was used to overlap two stratifications, based on 1970 
census data, of the SIPP Midwest region with the actual 
1980s design stratification for the SIPP Midwest region. 
(At the time of this test, 1990 census data was not yet 
available.) The 1970-based stratifications were produced 
by stratifying the 1980s SIPP noncertainty PSUs in the 
Midwest region using 1970 data. Both of the 1970-based 
stratifications partitioned the noncertainty PSUs into 
31 strata, using different sets of stratification variables. 
The stratifications based on 1980 and 1970 data were 
treated as ‘‘initial’’ and ‘‘final’’ stratifications for the 
purposes of the overlap algorithm. 

In the actual implementation, as noted in Section 3.1 
and detailed in Ernst and Ikeda (1994), a modification of 
the reduced-size procedure was used to overlap the 1990s 
SIPP design with the 1980s SIPP design, because the PSU 
definitions in the 1980s and 1990s designs were not iden- 
tical. The modified reduced-size procedure was used to 
overlap 103 final (1990s design) nonselfrepresenting strata 
in SIPP. 

The expected overlap was calculated for the reduced- 
size maximum overlap algorithm, for independent selection 
of final PSUs, and for an upper bound to the expected 
overlap for the optimal procedure. An upper bound was 
calculated instead of the actual optimal overlap, since the 
optimal overlap cannot be calculated for the larger strata. 
For the simulation, the upper bound used is the one stated 
in Section 3.3, w. + 2,, while for the production SIPP, 
a different upper bound, described in Ernst and Ikeda 
(1994), was required because the PSU definitions in the 
1980s and 1990s were not identical. 

The results from the two final stratifications in the 
simulation were generally similar to each other. Combining 
the results from both stratifications, the mean expected 
overlap for this set of 62 strata was 1.552, 1.569 and 0.480 
PSUs/stratum for the reduced-size procedure, the upper 
bound to the optimal overlap and independent selection 
respectively. For the actual SIPP implementation, the 
corresponding number was 1.523, 1.647 and 0.582, respec- 
tively, while the corresponding expected number of PSUs 
overlapped for the 103 strata was 156.9, 169.6 and 59.9, 
respectively. Thus, in both the simulations and the pro- 
duction SIPP, the reduced-size procedure yielded results 
reasonably close to the upper bound for the optimal 
procedure. 

The reduced-size algorithm took a fairly short time to 
run on most strata. The CPU times in the simulation for 
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final strata with different numbers of PSUs are given 
below. The reduced-size program was run on a Solbourne 
5/605 computer. The median number of PSUs in a 
stratum, for the entire group of 62 strata, was 17 PSUs. 
The 68 PSUs stratum was the largest stratum. 


Table 6 
CPU Times for Reduced-Size Procedure 


CPU Time 


Number of PSUs Pas ake 
(hrs:min:sec) 


18 0:36 
37 5:44 
49 24:05 
68 2:23:43 


We also calculated for the actual SIPP implementation, 
that of the 103 final strata overlapped by the modified 
reduced-size procedure, 41 would not have run under the 
optimal procedure. This calculation was based on our 
estimate that the maximum size transportation problem, 
in terms of number of variables, that could have run in 
production was 4 x 10°. The number of variables for the 
optimal procedure was less than 4 x 10° for all 56 strata 
for whichn < 14, but exceeded this limit for all but 6 of 
the 47 strata withn => 15, including two withn = 15. The 
maximal size of the transportation for the optimal proce- 
dure among the 103 strata occurred for a stratum with 
n = 46, for which there were 3.61 x 10! variables. In 
contrast, there were 1.03 x 10° variables for the modified 
reduced-size procedure for this stratum. 

Another question of interest is the overlap effectiveness 
of the reduced-size procedure in comparison with the 
overlap procedure of Ernst (1986). In general it is believed 
that the reduced-size procedure should produce a higher 
Overlap in situations when both are usable, since the 
reduced-size procedure makes use of the stratum-to- 
stratum independence in the initial design. However, 
although the procedure in Ernst (1986) is applicable to 
two-PSU-per-stratum designs, no computer program has 
ever been written at the Census Bureau (or anywhere else 
that the authors are aware of) to implement this procedure 
for such designs, since there has not yet been a production 
application for this program. Consequently, we cannot 
make a direct comparison of these two methods on the 
same data. However, a crude comparison can be made 
from the results of the reduced-size overlap procedure for 
SIPP data and the results of the overlap using the proce- 
dure in Ernst (1986) for the overlap of 1990s CPS and 
NCVS designs with their respective 1980s designs. (Both 
the 1980s and 1990s designs for CPS and NCVS are one- 
PSU-per-stratum designs.) 


For CPS, the overlap procedure resulted in an average 
increase in expected overlap, in comparison with indepen- 
dent selection, of .26 PSUs/stratum, and for NCVS the 
overlap procedure resulted in an average increase in 
expected overlap of .30 PSUs/stratum. This compares 
with an increase of .94 PSUs/stratum for the reduced-size 
procedure over independent selection for SIPP. If the two 
overlap procedures are equally effective, then one might 
expect that the increase in overlap per stratum for SIPP 
would be roughly twice as large as for CPS and NCVS, 
since SIPP has a two-PSUs-per-stratum design. By this 
standard, the reduced-size procedure program performs 
better than the procedure in Ernst (1986). However, since 
the stratifications were quite different for these three 
surveys, the validity of this comparison is open to question. 

For the example considered in Sections 2 and 3, a valid 
comparison of the different overlap procedures can be 
made, since the expected overlap values for the procedure 
in Ernst (1986)), 1.625, was easily calculated by hand. For 
the reduced-size procedure the corresponding overlap 
value is 1.725, and for the optimal procedure it is 1.735. 


CONCLUSIONS 


The reduced-size overlap procedure presented in this 
paper meets its two key objectives in practice. It reduces 
the size of the transportation problems to a usable size, 
as evidenced both by the size of the transportation problem 
in the formulation (3.1)-(3.3), and the fact that it has 
actually been implemented in the redesign of a major 
survey. In addition, the procedure accomplishes the size 
reduction while yielding nearly optimal overlap, at least 
for the SIPP application. It can only be used when the 
PSUs in the initial design are selected independently from 
stratum to stratum, but when this condition is met we 
believe it is the overlap procedure of choice for large strata. 
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How Prenotice Letters, Stamped Return Envelopes 
and Reminder Postcards Affect Mailback Response Rates 
for Census Questionnaires 


DON A. DILLMAN, JON R. CLARK and MICHAEL D. SINCLAIR! 


ABSTRACT 


In a 1992 National Test Census the mailing sequence of a prenotice letter, census form, reminder postcard, and 
replacement census form resulted in an overall mailback response of 63.4 percent. The response was substantially 
higher than the 49.2 percent response rate obtained in the 1986 National Content Test Census, which also utilized 
a replacement form mailing. Much of this difference appeared to be the result of the prenotice - census form — 
reminder sequence, but the extent to which each main effect and interactions contributed to overall response was 
not known. This paper reports results from the 1992 Census Implementation Test, a test of the individual and 
combined effectiveness of a prenotice letter, a stamped return envelope and a reminder postcard, on response rates. 
This was a national sample of households (n = 50,000) conducted in the fall of 1992. A factorial design was 
used to test all eight possible combinations of the main effects and interactions. Logistic regression and multiple 


comparisons were employed to analyze test results. 


KEY WORDS: Mail survey; Response rates; Multiple comparisons; Logistic regression. 


1. INTRODUCTION 


A decline of 10 percentage points from 75 to 65 in the 
mailback response rates for the 1990 U.S. Decennial 
Census has stimulated the conduct of research aimed at 
finding ways to improve response. Each percentage point 
gain in response has the potential for saving approximately 
$16 million in personal visit enumeration costs (Miskura 
1992). From an earlier experiment it was learned that 
respondent-friendly construction and asking somewhat 
fewer questions than posed in the 1990 Census short ques- 
tionnaire improved mailback response rates by 8.0 per- 
centage points (Dillman, Clark and Sinclair 1993). An 
experimental census form with these features was returned 
by 71.4 percent of households, compared to 63.4 percent 
of those which had received the 1990 Census short form 
as acontrol. Response rates for both of these forms were 
substantially higher than had previously been obtained in 
similar non-census year tests. For example, in the 1986 
National Content Test which utilized a questionnaire 
equivalent to the 1990 Census short form, a 49.2 response 
rate was obtained. It was hypothesized that part of the high 
response observed in the recent experiment was due toa 
multiple contact implementation strategy which consisted 
of a prenotice letter, a reminder postcard and a replace- 
ment questionnaire. 

The purpose of this paper is to report results of the 1992 
Implementation Test (IT), a test designed to determine the 
relative and combined contribution to mailback response 
of the prenotice letter and reminder postcard used in the 


previously reported experiment (Dillman ef a/. 1993). Also 
included in the test is the effect of including a stamped return 
envelope (vs. business reply) with the mailed census form. 

The 1990 U.S. Decennial Census required surveying 
over 100,000,000 households. Cost considerations alone 
suggest the importance of learning the extent to which each 
of these three response-inducing techniques might be 
employed in improving household response. Although 
past research has suggested that each of the three elements 
can be important to improving response, little information 
is available on potential interactions among them. The 
study was designed in such a way as to explore the extent to 
which their combined uses are additive and/or interactive. 


1.1 Past Research 


Numerous studies have confirmed that the most impor- 
tant determinant of overall response to mail surveys is the 
number of contacts (e.g., Scott 1961 and Heberlein and 
Baumgartner 1978). Both prenotices and reminders have 
been demonstrated as being effective promoters of response 
(e.g., Kanuk and Berenson 1975, Linsky 1975 and Fox et al. 
1988). However, past research has provided minimal insight 
into their relative importance as inducers of response. 

Past research is generally consistent in suggesting that 
inclusion of a stamped return envelope (vs. a business reply 
envelope) improves response (Scott 1961, Kanuk and 
Berenson 1975, Duncan 1979, Harvey 1987 and Fox et al. 
1988). A noteworthy exception is a regression analysis of 
previous studies by Heberlein and Baumgartner, which 
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found no significant effect for the inclusion of stamped 
return envelopes (1979). A review study by Armstrong 
and Luske reported 20 studies in which alternatives to 
business reply envelopes had been tested (1987). In each 
of these comparisons the absolute level of response to the 
alternative was significantly higher in 15 of the 20 cases, 
by an average of 9.2 percentage points. Six studies of 
metered marks vs. envelopes with real stamps were 
reported. On average they showed a 3.4 percentage point 
advantage for stamps. Finally, four studies in which a 
constellation of response inducing factors was used to 
insure high overall response rates showed a 2-4 percentage 
point advantage for stamped over business reply envelopes 
(Dillman 1978). 

The three response stimuli to be tested here are among 
the top eight techniques reported consistently in the research 
literature as factors which improve mailback response 
rates. Others include financial incentives, special postage, 
choice of sponsor, personalization and interest (or salience) 
(Dillman 1991). 

Two of these eight factors, financial incentives and 
special postage (e.g., certified or two day priority mail) 
were judged impractical for use in a census of more than 
100,000,000 households. A third factor, sponsorship by 
the U.S. Bureau of the Census, was considered desirable 
from the standpoint of encouraging response. A fourth 
factor, respondent interest, or question salience could not 
be manipulated in the sense that the survey questions are 
specified by federal laws. The fifth factor, personalization 
of correspondence was limited by the fact that Census 
forms cannot be addressed to individuals and are neces- 
sarily sent to only household addresses. By examining the 
individual and combined response effects of the prenotice, 
stamped return envelope and reminder, we hoped to learn 
whether the use of one or more of these elements would 
substitute for another, therefore making it possible to 
improve response at less cost. 


1.2 Design and Integration of Treatment Elements 


Certain features of the census form mailout packet 
suggest that it may be overlooked or ignored by those to 
whom it is sent. By necessity it is sent only to household 
addresses; names cannot be used to address any of the 
letters. Accurate processing of returned questionnaires 
requires identification of the household address on the 
questionnaire itself. Separately addressing an outside 
envelope, letter and questionnaire and being sure that the 
correct components are inserted into the appropriate 
envelope presents a serious quality control problem in a 
large census. Therefore it is considered important to print 
addresses only on one of the pieces that has to be merged 
together for the mailout package. Consequently, a win- 
dowed envelope through which the address on the ques- 
tionnaire can be seen is used to deliver it. 


The combined effect of the inability to use resident 
names plus size and outward appearance of the windowed 
envelope suggest that it contains unimportant material or 
perhaps, ‘‘junk mail.’’ Also, research on nonresponse to 
the 1990 Census revealed that some people did not recall 
receiving their census questionnaire in the mail, or saw it, 
but did not open it, both of which might have resulted 
from a mass mailing appearance (Kulka ef al. 1991). 

In this experimental test the prenotice letter and re- 
minder postcard were designed to bring attention to the 
envelope containing the census form. This was accom- 
plished in five ways. First, the prenotice was developed as 
a letter, and the reminder as a postcard. It was reasoned 
that people were more likely to look at two pieces of mail 
which appeared different from one another. The letter 
format was chosen for the prenotice in order to save the 
more convenient postcard format for the reminder. 

Second, the prenotice letter consisted of a letter from 
the Director of the Census Bureau with the notation ‘‘To 
the residents at’’ and the address imaged onto stationery 
in the normal inside address position. Our goal was to 
communicate that the census questionnaire which would 
soon arrive was specifically for people at that address. This 
address also doubled as an outside address, being visible 
through a windowed envelope, thereby avoiding the 
quality control concern noted for the census form mailing 
of merging separately addressed components. 

Third, the prenotice was scheduled to be delivered a few 
days before the envelope containing the census form itself, 
and the reminder was scheduled to arrive just a few days 
afterwards. The mailout dates were September 21st, 24th, 
and 29th, respectively. It was reasoned that to be effective, 
a reminder (without a replacement questionnaire) should 
arrive within a few days of the questionnaire, before 
normal household cleaning would have resulted in un- 
opened mail being thrown out. 

Fourth, the wording of the prenotice, ‘‘Within the next 
few days you should receive. ..’’ and the reminder, ‘‘A 
few days ago you should have received. . .”’ were designed 
to encourage recipients to look for the census form. Fifth, 
the use of the Director’s letterhead stationary and white 
postcard stock which showed the seal of the Department 
of Commerce above the reminder message, were aimed at 
communicating that the census questionnaire was from the 
government and not from some other group attempting 
to emulate a governmental appearance, as is sometimes 
done by noting, e.g., ‘‘this is your official notice.’’ 

The stamped return envelope’s positive influence, if 
any, On response may result from encouraging trust that 
the request is legitimate and important (otherwise why 
would the sender ‘‘waste’’ a stamp, which could be torn 
off and used for another purpose and/or a recipient’s 
reluctance to throw away something of value, i.e., an 
uncanceled stamp). The prenotice, and to some extent the 
reminder, could enhance the stamp’s effect by getting the 
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envelope containing the census form opened. Also, once 
opened, the awareness of an uncancelled stamp could 
discourage throwing away the contents so that the effect 
of the reminder is enhanced. 

In order for the prenotice, stamped return and reminder 
to mutually support one another, it was deemed important 
that first class mail be used. Had bulk rate been used, and 
the mailings been closely spaced, it was likely that in some 
households a later mailing would have arrived before an 
earlier one. 

In sum, this test involved more than simply juxtaposing 
three separate test elements from the literature. The 
elements were operationalized in ways that improved the 
likelihood that each would augment effects of the others, 
and be feasible for use in large scale mailings. Practically, 
we hoped to learn whether one or more of the elements 
might be eliminated without a significant loss of response, 
thus showing how to save costs for a census mailing. 


2. EXPERIMENTAL DESIGN 


A factorial design, consisting of all eight of the possible 
combinations of the three main effects, was used for the 
experiment. The treatments were as follows: 

. None (control), 

. Prenotice letter only, 

. Stamped return envelope only, 

. Reminder postcard only, 

. Letter plus stamped return, 

. Stamped return plus reminder, 

. Letter plus reminder, and 

. Letter plus stamped return plus reminder. 


ONNMN WN 


2.1 Sample Design 


The sampling universe consisted of all housing units in 
the questionnaire mailback areas identified by Census 
Bureau address files. The 449 district office (DO) areas for 


161 


the 1990 Census were selected as the geographic units for 
defining the strata for the test. Two strata were defined. 
Due to the high correlation between the minority rate 
(minority is defined as including all Black and Hispanic 
classifications) and the 1990 Census mail response rate, 
the stratification objectives were met by ranking the DOs 
by their percent minority. DOs with a combination of high 
minority (Black and/or Hispanic origin) population and 
low 1990 questionnaire mail response rates were defined 
as ‘‘low response areas’? (LRA) and made up the first 
stratum. The remaining DOs were classified as ‘‘high 
response areas’’ (HRA) and constituted the second stratum. 

The first stratum, consisting of 67 DOs, had a combined 
minority population of about 64 percent and encompassed 
about 11 percent of all housing units in the census mailback 
areas. The second stratum of 382 DOs had a combined 
minority population of about 15 percent. The HRA stratum 
had a cumulative mail response rate in the 1990 Census of 
approximately 10 percentage points higher than the LRA 
stratum. 

A sample of 50,000 housing units was selected with 
25,000 units in each stratum. The LRA stratum was over- 
sampled to concurrently study factors related to differential 
undercount, which falls outside the scope of this paper. 
Each stratum was divided into eight equally sized panels 
to test the eight different treatments. A systematic sample 
of 3,125 housing units was selected from each panel/ 
stratum combination. Once a housing unit was selected, 
the seven subsequent units were also selected. The resulting 
households in each of the eight unit clusters were randomly 
allocated to a panel. Hence, all eight neighbors got differ- 
ent treatments. The sample was clustered to reduce the 
sampling variance in the panel-to-panel comparison. 

The sample size selected for this study was developed 
by extensive data simulations which indicated that the 
50,000 unit sample would be sufficient for detecting a 
minimum of a 3 percent difference in all pairwise treatment 
comparisons. 


Table 1 
Implementation Test Final Rates National and Stratum Level Estimates 


Response Rate (%) Estimates and Standard Errors (%%) 


Treatment National 
Estimate Standard Error 

1. Control 50.0 0.8 
2. Prenotice Letter Only 56.4 0.8 
3. Stamped Return Envelope Only 52.6 0.8 
4. Reminder Card Only 58.0 0.8 
5. Letter and Stamp 59.8 0.8 
6. Stamp and Reminder 5955 0.8 
7. Letter and Reminder 62.7 0.8 
8. Letter, Stamp and Reminder 64.3 0.8 


1990 High Response Areas 1990 Low Response Areas 


Estimate Standard Error Estimate Standard Error 
51.9 0.9 36.3 0.9 
58.6 0.9 40.5 0.9 
54.5 0.9 37.9 0.9 
60.2 0.9 42.0 0.9 
62.1 0.9 43.0 0.9 
61.8 0.9 42.6 0.9 
65.0 0.9 45.4 0.9 
66.5 0.9 47.8 0.9 
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3. FINDINGS 

The major results from this study are presented through 
two analytical methods, first through multiple pairwise 
comparisons of treatment means and secondly through 
logistic regression. See Appendix for estimation proce- 
dures. Both methods provide consistent results. The overall 
response rates and standard errors for each of the treat- 
ments at the national and stratum levels are presented in 
Table 1. They range from 50.0 percent for the control 
group to 64.3 percent when all three main effects are 
applied together. 


3.1 Multiple Comparisons of Mail Response Rates 


Twenty eight comparisons are presented in Table 2 
corresponding to all possible pairwise comparisons of the 
8 treatments. Given the space restrictions in the table, the 
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following abbreviations were used: C = control, L = pre- 
notice letter, S = stamped return envelope, R = reminder 
postcard. 

The first three comparisons in Table 2 illustrate the 
improvements in response that main effect components 
added to response individually above and beyond the 
control treatment. The estimated improvement in response 
due to the prenotice letter was 4.2 percent in the LRA 
stratum, 6.7 percent in the HRA stratum and 6.4 percent 
at the national level. The estimated improvement due to 
the reminder card was 5.7 percent in the LRA stratum, 
8.3 percent in the HRA stratum and 8.0 percent at the 
national level. All of these improvements are significant. 
Thus, the principal finding of this study is that both the 
prenotice letter and the reminder card increased mail 
response at the national and stratum level. No significant 
improvements were noted for the stamped return envelope 
at the national or stratum level. 


Table 2 


Differences in Response Rates - Each Component in the Presence of Another Component 


Response Rate Differences (%) and 90% Confidence Intervals (C.I.) 


Experimental 


1990 Low Response Areas 


1990 High Response Areas 


Comparisons pabon2 (LRA) (HRA) 
Difference 90% C.I. Difference 90% C.1. Difference 90% C.1. 
ie goSxe 6.4 3.3 to 9.5* A) 0.9 to 7.5* 6.7 3.2 to 10.2* 
Mepsrocil es) —0.5 to 5.6 ey —1.7 to 5.0 Delf —0.8 to 6.1 
BaP RIC 8.0 4.9 to 11.1* Si 2.4 to 9.1* 8.3 4.9 to 11.7* 
AT ESa€ 9.8 6.7 to 12.9* 6.8 3.4 to 10.1* 10.2 Gal tOmlonieas 
5. SRE E 9.5 6.4 to 12.5* 6.4 3.0 to 9.7* 9.9 6.5 to 13.3* 
6. LR-C 1DF7, OCtOMS 7s 9.2 Syaten UO) PAY 13.2 9.7 to 16.6* 
7. LSR-C 14.2 IDS A2 Tie) 1/2 WS) 8.2 to 14.8* 14.6 11.3 to 18.0* 
8. L-S 3.8 0.8 to 6.9* DS —0.9 to 5.9 4.1 0.6 to 7.5* 
9, R-L 1.6 —1.5 to 4.8 eS —1.9 to 5.0 1.6 — 1.96 to 5.10 
10. R-S 325 2.4 to 8.5* 4.1 OM/MowES = 5.6 2-2, to.9.0* 
ik IES SAG 3.4 0:3 to6:5* 2.6 —0.9 to 6.0 35) 0.03 to 7.0* 
We SR = IL Boll 0.03 to 6.2* DP —1.3 to 5.6 3.2 — 0.3 to 6.6 
ley IRS =e 6.3 Bie OSS 0) 1.5 to 8.4* 6.4 3.0 to 9.9* 
14. LS-S cs 42 to 10°3* So SI TI@) fe305" 7.6 4.1 to 11.0* 
15. SR-S 6.9 3.8 to 10.1* Ax] eZ torse2s UD! 3.8 to 10.7* 
16. LR-S 10.1 Hmletomise = eS AaitomieO* 10.5 7.0 to 13.9* 
NG, dls 8 1.8 —1.3 to 4.9 ill —2.4 to 4.5 1.9 —1.6to 5.4 
18. SR-R M8) —1.6 to 4.5 ON —2.8 to 4.1 1.6 —1.8 to 5.0 
19. LR-R 4.7 1.6 to 7.7* 85 — 0.02 to 6.9 4.9 VStolsso 
20ESRI 7.9 4.8 to 10.9* he 3.9 to 10.7* 7.9 Assitonilea* 
Dive SRetS Viley 8.7 to 14.7* 9.8 6.4 to 13.3* 12.0 8.6 to 15.4* 
2 IESREN 6.2 3.2 to 9.3* 5.8 2.3 to 9.3* 6.3 2.9 to 9.7* 
25a SRS 4.4 WS Foy Waste 4.7 ez LOnS ea 4.4 LOOP 58% 
24. LSR - SR 4.8 Letom ess Soll 1.7 to 8.6* 4.7 ESTOS = 
Daw S Rea ke 1.6 —1.4to 4.5 Pap) —1.1 to 5.8 115) —1.8 to 4.8 
26. SR- LS —0.3 —3.3 to 2.7 —0.4 — 3.8 to 3.1 —0.3 —3.7 to 3.1 
ZIM ERI= LS 2.9 —0.2 to 6.0 2.4 —1.1 to 5.9 2.9 — 0.6 to 6.4 
23. sURI='SR 30 0.2 to 6.2* 2.8 —0.6 to 6.2 33 —0.1 to 6.6 


A C.I. marked with an * indicates the difference was statistically significant at « = .10(9-in-10 chance that the C.1.s will include the actual differences). 
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3.2 Logistic Regression Analysis 


A model including components for the stratum, pre- 
notice letter, stamp and reminder card including all of the 
interaction terms was evaluated. Modeling was also per- 
formed at the stratum level using only parameters for the 
component effects and their interactions. 

The results of the full model analysis indicate that only 
the main effects of the letter and the reminder card along 
with the intercept and stratum term are statistically signifi- 
cant in the model. Given these results, additional modeling 
at the national level was accomplished with a reduced 
model including only the stratum main effect, the indi- 
vidual components and the component interactions. The 
results of this modeling are presented in Table 3 below. 


Table 3 


Analysis of Weighted Least Squares Logistic Regression 
Modeling Reduced Model, no Stratum by 
Component Interactions 


Estimated Parameters and 90% 
Bonferroni Confidence 


Model Parameters Intervals (C.I.) 


Estimate 90% C.I. 
Intercept, Bo — .61 — .686 to — .545* 
Stratum, 6; .738 .689 to .789* 
Letter, Bz LH .130 to .324* 
Stamp, 83 .090 — .006 to .186 
Reminder, 64 291 .194 to .387* 
Letter/Stamp, 65 .036 —.101 to .173 
Letter/Reminder, 6¢ — .054 — .192 to .083 
Reminder/Stamp, 67 — .043 — .179 to .093 
Let/Reminder/Stmp, Bg — .003 —.197 to .191 


A C.I. marked by an * indicates the difference was statistically significant 
aton— 105 


The results of both modelings show that significant 
improvements were realized from the prenotice letter and 
reminder post card, but not from the stamped return 
envelope for the national and within stratum models. 
These results correspond to those presented by the multiple 
comparisons above. None of the interaction terms were 
statistically significant, indicating the effect of the compo- 
nents are basically additive in nature. 


4. DISCUSSION AND CONCLUSIONS 


The prenotice letter, stamp and reminder postcard 
individually improved response rates by 6.4, 2.5 and 8.0 
percentage points, respectively. The increase of 2.5 was 
not statistically significant. The effects of the elements 
were also found to be mostly additive, and did not interact 
with one another. In comparison to the control group, the 
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combination of letter-stamp improved response 9.8 per- 
centage points, the stamp and reminder, 9.5 percent, and 
the letter and reminder, 12.7 percent. All three elements 
together improved response by 14.3 percent. Each use of 
the letter and reminder added significantly to response, but 
the stamp only added significantly when used with a pre- 
notice and no reminder. The most important conclusion 
from this experiment was that both the prenotice letter and 
reminder postcard are important to achieving a high 
response and that neither eliminates the effect of the other. 

Although the individual effect (2.5 percent overall) of 
the stamped return envelope is slightly smaller than needed 
for significance, it is of similar magnitude to what has been 
found significant in past research (Armstrong and Luske 
1987; Dillman 1978, 1991). In light of the preponderance 
of past research showing its effectiveness, this technique 
should probably not be completely dismissed as being inef- 
fective. It also appears that the stamped return envelope 
relates differently to the prenotice and reminder. When 
used alone with the prenotice, the effect of the stamped 
return is significant (3.4 percentage points), but it is clearly 
insignificant (1.6 percentage points) when a reminder is 
included in the mailout procedures. The reminder compen- 
sates for the lack of a stamped return envelope, whereas 
the prenotice appears to amplify its effect. It may be that 
a prenotice alerts people to notice and open the census 
form mailout package, and once opened, people are then 
encouraged to respond by the presence of the stamped 
return envelope. This differential connection to the 
mailings that precede and those that follow, appears not 
to have been examined in past research. A practical impli- 
cation for the Census is that if a prenotice letter and no 
reminder is used, a stamped return envelope might add 
significantly to response, but be of less importance if a 
reminder postcard is used, as was done in the last census. 

There are at least two significant barriers to the direct 
application of this research to conduct of the 2000 Census. 
First, it is important to recognize that these tests are being 
done in non-census years. In the past the Census Bureau 
has obtained much lower response rates in non-census 
years than in census years. For example, the 1986 National 
Content Test, obtained only a 49.2 percent response 
employing a replacement questionnaire, while the 1990 
Census without employing a replacement questionnaire, 
achieved a 65 percent response rate. The usual explanation 
for this difference is ‘‘census climate,’’ a succinct explana- 
tion of the combination of media attention, advertising, 
and cultural sense of participation that seems to build each 
decade during the census year. 

The response rates obtained in our tests with the use of 
the five elements found to increase response are much 
higher than normally obtained in non-census years, but 
are close to the same, or perhaps a little lower, than those 
obtained during the last decennial census when none of 
these elements were used. We do not know whether the 
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existence of a ‘‘census climate’’ will substitute for the 
effects of these elements or add to the response likely to 
be obtained in a census year. Certainly a 30 percentage 
point increase will not be realized in the 2000 Census since 
that would suggest a response of nearly 100 percent. 
Therefore, considerable uncertainty remains with respect 
to the exact implications of the present findings for the 
2000 Census. 


APPENDIX 
Estimation Procedures 


Analytical results are derived from two separate 
methods, multiple comparisons among the mail response 
rates by treatment group, and logistic regression analysis. 
Each method has advantages over the other in terms of 
ease of interpretation and ease of statistical inference; 
hence a combined approach was utilized to bring forth the 
best of both methods for presentation. 

The national mail response rate estimates for a given 
panel as presented in this study is computed by dividing 
the weighted total of the number of questionnaires returned 
by the weighted total number of forms mailed out less 
weighted postmaster returns (mostly vacant units). 

Multiple comparisons of the 8 treatment mail response 
rates were reviewed to determine the level of increase in 
the mail response to each of the treatments. These com- 
parisons involved a pairwise assessment of each of the 
treatments with the control panel and with each other. 

The logistic regression procedures provide a quick and 
effective means for evaluating whether or not observed 
increases from each of the components, especially inter- 
actions, are the result of sampling variation or imply a true 
increase, and if these increases are influenced by the 
presence of other components. However, parameter esti- 
mates cannot be easily equated to the mail response rates. 
A detailed overview of the logistic regression methodology 
is provided in Thompson 1993. 

Response rates were calculated for each of the treatment 
groups within stratum and at the national level (stratum 1 
and stratum 2 combined). Standard errors for the national 
estimates were computed using the stratified jackknife 
variance procedure (Wolter 1985). The estimates were pro- 
duced by the VPLX< statistical software package. Standard 
errors for the within stratum estimates were computed 
using the formula for the simple random sampling jack- 
knife variance procedure. 

The primary analysis involved pairwise comparisons of 
the differences between response rates for eight treatments, 
both overall at the national level and for the two strata, 
LRA and HRA. 

Because of the various hypotheses being tested, all 
possible pairwise comparisons (28 total) between the eight 
treatments are analyzed in the experiment. In the logistic 


regression framework 8 or more model parameters are 
tested for significance. The more comparisons that are 
made, the greater the potential that some of these compar- 
isons will be incorrectly declared significant. In this case, 
additional statistical measures are employed to control the 
overall error of the decision process. 

The analysis has been carried out so that statements 
about the entire ‘‘family’’ of 28 pairwise comparisons or 
the logistic regression parameters are made while main- 
taining the 90 percent (a Census Bureau standard) con- 
fidence level simultaneously for all comparisons. All 
90 percent confidence intervals for the pairwise compar- 
isons were adjusted using Dunnett’s C-procedure for 
comparing pairwise contrasts of the test panel estimates 
(Hochberg and Tamhane 1987). Bonferroni simultaneous 
inference procedures were used to evaluate the statistical 
significance of the logistic regression parameters. 
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Consistency of Census and Vital Registration Data 
on Older Americans: 1970-1990 


LAURA B. SHRESTHA and SAMUEL H. PRESTON! 


ABSTRACT 


Major uncertainties about the quality of elderly population and death enumerations in the United States result from 
coverage and content errors in the censuses and the death registration system. This study evaluates the consistency 
of reported data between the two sources for the white and the African-American populations. The focus is on the 
older population (aged 60 and above), where mortality trends have the greatest impact on social programs and where 
data are most problematic. Using intercensal cohort analysis, age-specific inconsistencies between the sources are 
identified for two periods: 1970-1980 and 1980-1990. The U.S. data inconsistencies are examined in light of evidence 
in the literature regarding the nature of coverage and content errors in the data sources. Data for African-Americans 
are highly inconsistent in the 1970-1990 period, likely the result of age overstatement in censuses relative to death 
registration. Inconsistencies also exist for whites in the 1970-1980 intercensal period. We argue that the primary 
source of this error is an undercount in the 1970 census relative to both the 1980 census and the death registration. 
In contrast, the 1980-1990 data for whites, and particularly for white females, are highly consistent, far better than 
in most European countries. 


KEY WORDS: Age misreporting; Coverage; Mortality; Census evaluation; Death registration; Data quality; 


Mortality crossover, United States. 


1. INTRODUCTION 


Conventional methods of estimating levels of mortality 
in more developed countries use data from two different 
sources. The numerators of death rates are normally 
counts of deaths derived from vital statistics. The denomi- 
nators are usually derived from census counts of persons 
alive. The accuracy of calculated rates depends on the 
quality of data from both sources. 

This paper reports the results from a test of data quality 
applied to United States data for two intercensal periods: 
1970-1980 and 1980-1990. In particular, we examine the 
consistency of reported changes in the size of a cohort 
between two censuses and the recorded number of inter- 
censal deaths for that cohort, with allowance for inter- 
censal cohort migration. All data refer to the population 
in single years of age and separate tests are conducted for 
the black and white populations. 

Our focus is on the older population (aged 60 and 
above), where mortality trends have the greatest impact 
on social programs (Preston 1993) and where data quality 
is most problematic. The white population of the United 
States appears to have lower death rates above age 80 than 
any other industrialized country (Vaupel 1993). If valid, 
this comparison would have important implications for 
evaluating the relative quality of medical systems. But the 
African-American population of the United States has 
even lower rates than the white population above age 80, 


reflecting the well-known crossing over of the age patterns 
of mortality between the races somewhere between ages 
75 and 85. Whether either set of mortality rates can be 
accepted at face value depends, of course, on the quality 
of the data. Data on blacks has elicited considerable skep- 
ticism (e.g., Zelnik 1969; Coale and Kisker 1990), although 
most observers appear to accept the validity of the cross- 
over (Manton ef al. 1986; McCord and Freeman 1990). 

In the process of constructing new model mortality 
patterns for low mortality countries, Condran, Himes, 
and Preston (1991) report similar data quality tests for 
68 intercensal periods in 18 industrialized countries. In 
general, consistency was very good for cohorts aged 65 at 
the second census (66 of 68 data sets passed the consistency 
check). Consistency deteriorated with age; only about half 
of the data sets showed consistency at age 85 and fewer 
than 15% did so at age 95 (Condran et al. 1991: Table 7). 
The United States was not among the countries included 
in these tests because it lacked published data on deaths 
by single year of age. We are now able to fill in this impor- 
tant gap because we have processed data tapes on each 
individual death registered in the United States from 1970 
through 1988. (The single year death distribution for 1989 
(full year) and 1990, January to March only, is estimated 
using published group data from the National Center for 
Health Statistics and the 1988 single-year death distribu- 
tion. Details are provided in Appendix A.) These tapes are 
produced by the National Center for Health Statistics 


! Laura B. Shrestha, The World Bank, Human Development Department, 1818 H Street, N.W., Washington, DC 20433, U.S.A.; Samuel H. Preston, 
Population Studies Center, University of Pennsylvania, 3718 Locust Walk, Philadelphia, PA 19104, U.S.A. 


168 Shrestha and Preston: Consistency of Census and Vital Registration Data on Older Americans: 1970-1990 


(NCHS) and are distributed by the Inter-University Con- 
sortium for Political and Social Research. For the years we 
have included, they contain approximately 50 million deaths. 


2. STUDY POPULATION AND DATA 


2.1 Background 


Three major sources of data are utilized: (1) national- 
level census enumerations from the U.S. Bureau of the 
Census for the years 1970, 1980 and 1990; (2) annual death 
registration data produced by NCHS; and (3) unpublished 
estimates of net immigration obtained from the U.S. 
Bureau of the Census. While the data sources are described 
in more detail in Appendix A, a brief description of the 
data and significant adjustments is warranted. 


2.2 The Census Enumerations 


We utilize census tabulations which are classified by 
race (black/white), sex, and single years of age (open- 
ended at age 100). The tabulations refer to the resident 
population of the 50 states and the District of Columbia. 
Included in the enumerations are: the institutionalized 
population, Americans travelling abroad temporarily, and 
foreign citizens having their usual residence (legally or 
illegally) in the United States (except foreign military and 
diplomatic personnel). Specifically excluded are: Americans 
overseas for an extended period and foreign citizens tem- 
porarily visiting the U.S. The official statistics do not 
adjust for census undercount, e.g., the failure to find and 
enumerate legal residents and undocumented resident 
immigrants. 

The term ‘‘resident population’’ implies that both 
the legal population and undocumented immigrants are 
included in the census tabulations. While undocumented 
persons were residing in the U.S. at the time of the 1970 
census, it appears that only a negligible number were 
counted. Hence, the legal resident population approx- 
imated the total resident population in the 1970 census. 
In the 1980 count, however, the U.S. Bureau of the Census 
estimates that, for the first time ever, a significant number 
of undocumented persons were enumerated. Estimates 
indicate that the count equalled 2.06 million undocumented 
persons. Of this number, in the age group 60 and above, 
10 thousand white males were enumerated; 19 thousand 
white females, 3 thousand black males, and 6 thousand 
black females (U.S. Bureau of the Census 1988). 

The official 1970 census tabulations are known to 
contain errors, the most conspicuous of which is the gross 
overstatement of the number of persons aged 100 years or 
more. Although the census enumerated 106,000 persons 
in this age group, indirect demographic estimates indicated 
that the correct centenarian count should have been in the 
range of 3,000 to 8,000 with a preferred estimate of 4,800 


(Siegel 1974; Siegel and Passel 1976; U.S. Bureau of the 
Census 1974). We utilize unpublished U.S. Census Bureau 
tabulations of the 1970 census, which correct for the 
centenarian overcount. Use of the corrected estimates is 
justified by two conditions: first, without adjustment, the 
excess is large enough to bias results at the oldest ages. 
Second, it appears that the overcount was not due to 
systematic misreporting of age into the centenarian popu- 
lation. Rather, it was the result of misunderstanding of the 
census form wherein individuals confused the columns 
intended for month of birth and year of birth (Siegel and 
Passel 1976). 

In both the 1980 and 1990 censuses, a large number of 
individuals enumerated chose to write in a response to the 
race question as opposed to selecting one of the specified 
all-inclusive race categories. For the total population, 
6.8 million individuals, largely of Spanish-origin, were 
affected in 1980, whereas the number increased to 9.3 million 
in the 1990 census. The official census tabulations are not 
directly comparable with other data sources since only the 
census enumerations contain a residual race category. To 
allow comparison with other data systems, the Census 
Bureau modified the 1980 and the 1990 enumerations to 
conform to historical categories of the racial groupings. 
The 1990 modification at the Census Bureau also involved 
“‘correction’’ for an age-related problem (for details, see 
Word and Spencer 1991). The decision was made to use 
the race-modified statistics for 1980 and 1990 from the 
Census Bureau for this research. The choice is justified by 
the sheer magnitude of individuals that would be excluded 
by use of the unmodified data, particularly for the white 
population. 


2.3 Death Registration Data 


The U.S. death registration data represent every death 
registered in the 50 states and the District of Columbia, 
classified by race, sex and age (single years of age to 
125+). To insure comparability with the census data, 
deaths of nonresidents of the United States (nonresident 
foreign nationals and U.S. nationals residing abroad) have 
been excluded. 

Adjustment is made for neither under-registration of 
deaths nor for misreporting of characteristics on the death 
certificates. Two problems were identified that affected 
the utilization of our intended intercensal methodology. 
The intercensal period covers the interval from April 1 to 
March 31, whereas the death registration data refer to 
calendar years. And both the death registration and the 
U.S. censuses’ data are reported by age at last birthday 
rather than by year of birth. We manage both problems 
by assigning deaths to triangles of time-age that correspond 
to ‘‘census years’’ beginning on April 1. For example, 
deaths reported in the one year interval between census 
date April 1, 1970 and April 1, 1971 to those aged 60 (last 
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birthday) at the time of the census can be classified into 
four categories: (1) deaths to those aged 60 in calendar year 
1970; (2) deaths to those aged 60 in calendar year 1971; 
(3) deaths to those aged 61 in calendar year 1970; and 
(4) deaths to those aged 61 in calendar year 1971. Using 
data on the date of death from the NCHS tapes, we 
assigned deaths to triangles of time-age that corresponded 
to the census year beginning on April 1, 1970. In doing so, 
we assume that deaths within each triangle are evenly 
distributed. This assumption is necessitated by the lack of 
reliable birth data for most of the cohorts considered in the 
paper, data that could be used to apportion deaths more 
accurately among adjacent birth cohorts. For a more detailed 
description of the methodology, see Shrestha 1993. 


2.4 Net Immigration Statistics 


We utilize unpublished net immigration statistics 
obtained from the U.S. Bureau of the Census. While the 
quality of net immigration statistics in the U.S. is widely 
acknowledged to be suspect (Hill 1985), the estimation of 
population size at the older ages is quite robust to variations 
in estimates of intercensal migration. This robustness 
results both from the smaller flow of net migrants at the 
older ages relative to younger ages and from the greater 
magnitude of deaths as a source of decrement in the older 
age groups relative to changes as a result of net migration. 
For instance, the net immigration data list an inflow of 
64 black males for the cohort aged 75 and above (in 1970) 
during the 1970 to 1980 decade. For comparison, over 
141 thousand deaths were recorded for the same cohort. 

Estimates of the flow of undocumented residents are 
not included in the constructed net immigration series, but 
will be considered in the interpretation of results. Their 
exclusion was precipitated by a number of factors. Esti- 
mates of the size and age-sex distributions of the illegal 
alien population vary widely due to insufficient data 
collection instruments in the U.S. But even the most 
exaggerated estimates of the number of undocumented 
migrants are minuscule relative to deaths at the older ages. 

We have described a number of adjustments that we 
have made to the basic data: use of unpublished 1970 
census tabulations because of a gross overcount of the 
centenarian population in the official statistics, use of race- 
modified tabulations of the 1980 and 1990 census, and 
exclusion of estimates of the undocumented alien popula- 
tion. In order to judge the effect of these adjustments on 
our results, we carried out numerous sensitivity analyses 
using uncorrected data. While only modest differences 
were observed between the results using official statistics 
and those with corrected tabulations (except at ages 100 
and above), the intercensal cohort analyses using uncor- 
rected data generally produced greater deviations in our 
final results, implying the overall appropriateness of these 
corrections. 
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3. SOURCES OF ERROR IN CENSUSES 
AND DEATH DATA 


Errors in demographic data have been classified into 
coverage errors and content errors. Coverage refers to 
the completeness with which persons or events that fall 
within the defined universe of a particular data system are 
recorded. Content refers to the quality of information 
about the persons or events that are in fact recorded. 
Either type of error in any data source can create incon- 
sistencies between intercensal change in cohort size and 
intercensal deaths. However, if both censuses and death 
registration suffer from the same net omission rate, then 
the sources will be consistent with one another; but under 
these circumstances, recorded death rates will also be 
accurate. 

Identical patterns of age misreporting in censuses and 
death registration will not, in general, produce consistency 
between changes in cohort size between the censuses and 
recorded numbers of intercensal deaths. The reason is 
that, because death rates rise with age, the age distribution 
of deaths at older ages is older than the age distribution 
of population. For example, if 10% of both persons and 
deaths at true ages 75-79 are misreported into the age 
interval 80-84, then the proportionate impact on population 
counts will be greater than the proportionate impact on 
death counts. Such a pattern of age misreporting would 
distort death rates, and would also be visible in the con- 
sistency tests that we apply. 

The Census Bureau has used demographic and statis- 
tical procedures to estimate the completeness of census 
coverage. Demographic procedures compare estimates of 
the true numbers of births minus estimated cohort deaths 
and migrations to census counts (see the summary in 
Robinson et a/. 1993 and Himes and Clogg 1992). Statis- 
tical procedures match a group of individuals identified 
in an alternative data source (such as the Current Popula- 
tion Survey) to individual-level records from the Census. 
A third approach is to compare the Census count of older 
persons to the count of individuals in Medicare files. 

A number of general conclusions for the old-age popu- 
lation were reached in the evaluation studies of the 1970 
census undertaken by the U.S. Bureau of the Census 
(1973, 1974, 1975). First, the magnitude of net error 
(combination of coverage and content errors) in the old- 
age statistics is greater than for the younger population. 
Second, females exhibited higher net error rates than 
males, largely the result of higher levels of age misreporting. 
But, gross omission rates (which are only one component 
of net error) were higher for males. Third, levels of net 
error, of gross omission, and of misreporting of demo- 
graphic characteristics are considerably higher for the U.S. 
black population than for the white. Fourth, the evidence 
suggests that considerable age misreporting exists in the 
official statistics. For example, it is interesting to note that, 
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for all four race-sex groups at ages 65-69 years in 1970, 
the estimates derived by demographic analysis suggest net 
census Overcounts, whereas the Medicare linkage study 
found gross census omissions in the magnitude of 2.1% 
(for white females) to 12.6% (for males of the black and 
other races category). This comparison implies that, while 
these groups have gross omissions in the number of 
persons enumerated at ages 65-69, other larger errors 
(presumably, especially age overstatement among persons 
below age 65) are operating in the other direction to inflate 
the net overcount estimates at these ages. One implication 
is that the characteristics of a substantial part of the 
population reported as 65 and over in the Census relate 
to persons who are in fact under age 65 (U.S. Bureau of 
the Census 1976). 

Relative to the 1970 census, the net error rates in 1980 
in most of the age-race-sex groups were significantly 
lower. As noted by the Bureau of the Census (1988), 
however, results from the Post Enumeration Program 
(PEP) and from the 1980 Housing Unit Enumeration 
Duplication study affirm that a considerable proportion 
of the total census count, likely in excess of 1.1%, repre- 
sented duplicate enumerations of individuals already in 
the census. Evidence implies much lower levels of duplica- 
tion in earlier censuses. Thus, ‘‘regrettably, duplication 
receives dubious credit for part of the improvement in 
1980 in net census coverage’’ (U.S. Bureau of the Census 
1988:10). 

The Census Bureau plans exhaustive evaluations of the 
quality of the 1990 Census, but the release of such analyses 
has been fragmentary to date. It does appear that the gross 
undercount was lower in 1980 than in 1990 (Robinson 
et al. 1993), but this may be the result of a higher degree 
of duplications in the 1980 census. A number of general- 
izations can be made regarding the pattern of net under- 
count in the 1990 census for the aged population. First, 
following its historical trend, the net error estimates for 
African-Americans surpass those of whites by a wide 
margin. The largest differential is noted for males aged 
60-64. The net undercount rate for black males equals 
10.3 percent, surpassing the white male estimate of 
2.6 percent by 7.7 percentage points. Second, whereas 
undercounts are observed for all of the male aged cate- 
gories, overcounts are noted in many of the female groups. 
Finally, as noted by Robinson et al. (ibid), the net coverage 
patterns are generally consistent across the last three 
censuses for each race-sex group. 

Official death statistics produced by the National 
Center for Health Statistics are the basic source of annual 
mortality data in the United States. The figures are gen- 
erally utilized without adjustment for underregistration or 
for misreporting of characteristics on the death certificate. 
It is generally assumed, however, that the death registration 
system is practically complete (Wilkin 1981; U.S. Bureau 
of the Census 1984a; National Center for Health Statistics 


1968) although no national test of its comprehensiveness 
has been conducted since the completion of the Death 
Registration Area in 1933. This assumption is based on the 
strict legal requirements for registration as well as on the 
needs of survivors for proof of death in connection with 
burial, settling estates and collecting insurance benefits 
(U.S. Bureau of the Census 1984a; Wilkin 1981). Calcula- 
tions by Coale and Kisker (1990), however, suggest that 
underregistration of deaths exists, particularly at the older 
ages. For the nonwhite population, for instance, registered 
deaths were 7% fewer than Medicare deaths for the male 
population aged over 80 in 1980, whereas registered female 
deaths were 10% fewer. These numbers, however, may be 
reflective of differential age reporting between the two 
sources, rather than of underenumeration. 

The best evidence regarding the consistency of age 
reporting between censuses and death registration - un- 
doubtedly the most important source of content error 
affecting our consistency test - matched a sample of death 
certificates from May to August 1960 with the 1960 census 
records (NCHS 1968; Hambright 1969). Although the data 
were collected before the time frame considered for this 
project, the study’s findings provide insight into what may 
be a continuing pattern of biases present in the census and 
death statistics. The authors found: (1) for whites, there 
was fairly high agreement between the sources even with 
increasing age — for nonwhites, however, there was less 
agreement; (2) in the event of disagreement, age discrep- 
ancies for the white population between the sources were 
generally within one year —- for nonwhites, however, the 
typical difference was more than one year, particularly at 
ages 45 and above; and (3) for whites of all ages and non- 
whites aged less than 45 years, the age reported on the 
death certificate was typically older than that reported on 
the census - for nonwhites aged 45 and above, however, 
age reported on the death certificate was, on average, 
younger than on the census. 

This study was unable to ascertain which data source, 
if either, provides the ‘‘true’’ age. To this end, Rosenwaike 
and Logue (1983) attempted to verify age reporting on the 
death certificate for the population aged 85 and over in 
the 1968 to 1972 period. The authors selected a sample of 
death records from those filed for decedents of extreme 
age in Pennsylvania and New Jersey. They then linked the 
individual who died to the 1900 manuscript census of 
population. A total of 1429 decedents were linked of 
whom 960 were white and 496 were non-white. 

They found that age agreement of matched census 
records with death certificates decreased as age increased 
for both racial groups. Striking differences were noted 
between racial groups. Agreement levels for whites were 
high, except at ages 100 and over. For nonwhites, however, 
significantly lower agreement was found. The authors 
further note that, within race, there was little difference 
by sex in agreement on age. 
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4. AN INTERCENSAL METHODOLOGY TO 
EVALUATE THE QUALITY OF 
OLD-AGE STATISTICS 


This analysis examines the extent of inconsistency in 
old-age U.S. data sources using an intercensal cohort 
methodology. The expected size of an open-ended age 
cohort in the second census can be estimated from its size 
at the first census and the intercensal deaths occurring to 
that cohort, after adjustment for migration (Condran, 
Himes and Preston 1991). Use of an open-ended category 
allows observation of the ratio trend while dampening 
error-induced extreme values at particular ages. It is insen- 
sitive to any errors of age reporting in deaths or population 
that occur within the population above the age that begins 
the open-ended age interval. 

Using census enumerations and death and migration 
statistics for an intercensal period, intercensal cohort 
analysis allows us to estimate the expected size of each 
open-ended age cohort in the subsequent census. The 
previously mentioned statistics, classified by single years 
of age, by sex, and by two races (white, black), were 
utilized to calculate the following equation for the expected 
population at the time of the second census: 


N,(2) = Ny-10(1) — Dy-10(1) + Meo) (D 


where 

N,(2) = the predicted population aged x and above 
at the second census, taken 10 years after 
the first. 


N,-19(1) = the enumerated population aged x — 10 
and above at time 1, the first census. 


D,.-\9(1) = the intercensal deaths which had occurred 
to the cohort aged x — 10 and above (at the 
first census). 


M,,_\o(1) = intercensal net legal immigration into the 
cohort aged x — 10 and above (at the first 
census). 


Similarly, the expected population at a given age (as 
opposed to at age x and above) can be calculated in an 
analogous manner. In either circumstance, the ratio of 
the observed population, enumerated in the subsequent 
census, to the expected population, can then be calculated 
(after simplifying the notation and assuming net migration 
to be zero) as: 


N,(2 
x (2) (2) 
Neo) =D 


x 


The change in the size of the cohort as measured at two 
successive censuses can be produced only by death or 
migration. A ratio of 1.00 would indicate complete con- 
sistency among the data sources. (Note that a ratio of 1.00, 
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while highlighting consistency, does not assure accuracy. 
On an individual level, for instance, if a person’s age was 
consistently overstated by n years, the method would fail 
to capture the misreporting.) In fact, however, the reported 
count will also be affected by: (1) coverage errors in either 
or both censuses; (2) under- (or over-) enumeration in the 
death registration data and/or the immigration statistics; 
and (3) misreporting of characteristics (age, race, efc.) in 
any or all of the data sources (Ewbank 1981; Shryock and 
Siegel 1976; Condran et a/. 1991). The ratio of observed 
to expected population is a useful diagnostic tool if patterns 
of deviation from 1.00 can be interpreted in terms of these 
underlying data errors. It is not a highly precise tool 
because different forms of error can produce the same 
pattern of ratios. Nevertheless, it can help discriminate 
among competing alternatives. 


5. HOW PATTERNS OF ERROR WILL AFFECT 
OBSERVED/EXPECTED RATIOS 


Effects of certain types of error are visible directly in 
the formula for the ratio itself (and have been confirmed 
by simulations that we have performed). To simplify the 
exposition, define R, in equation (2) as the ratio of ob- 
served to expected population for age x + at the second 
census. The following major possibilities for coverage 
error, and their implications for the age-pattern of ratios, 
can be distinguished: 


1) If N,—j9(1) and D are equally complete and N,(2) 
has a relative completeness level of C(2), then the age 
pattern of ratios will be constant with age and its level 
will be C(2). 


2) If N,(2) and D are equally complete and N,_ ;9(1) 
has a relative completeness level of C(1), then the age 
pattern of ratios will be: 


a) Above 1.00 and rising with age if C(1) < 1.00 
b) Below 1.00 and falling with age if C(1) > 1.00. 


The reason why an age trend in R, results from this 
pattern of error is that a particular proportionate error in 
N,—109(1) creates increasingly larger proportionate errors 
in the denominator as the two offsetting terms (one 
positive and one negative) in the denominator grow more 
equal in absolute value. This equalization occurs because 
a higher fraction of each cohort dies during the intercensal 
period as age advances. 


3) If Ny_-19(1) and N,(2) are equally complete and D 
has a relative completeness level of C(D), then the age 
pattern of ratios will be: 

a) Above 1.00 and rising with age if C(D) > 1.00 
(i.e., if completeness of death registration exceeds 
the completeness of enumeration in both censuses). 

b) Below 1.00 and falling with age if C(D) < 1.00. 
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Once again, an age trend is introduced because an equal 
proportionate error in D will create larger proportionate 
errors in the denominator as its two components become 
more equal in absolute value. 

Some of the effects of age misreporting patterns can 
also be understood by examining the components of this 
formula. Shrestha (1993) and Condran et al. (1991) intro- 
duce various errors into simulated errorless data sets 
typical of the current demographic conditions of the 
United States and the Netherlands respectively. They show 
that a pattern of net overstatement of age that is confined 
to the two censuses will produce a pattern of ratios that 
hovers around 1.00 until advanced ages, whereupon it falls 
to very low values. The reason why the ratio declines below 
1.00 is, once again, that an error in one component of the 
denominator (in this case, inflation of N,_j9(1) by age 
overstatement) introduces disproportionate effects in the 
denominator. Even though the rapid tapering off in the 
age distribution can result in N,(2) being more inflated 
than N,_ ;9(1), eventually the inflation of the denominator 
exceeds that of the numerator and the ratios fall. (For an 
illustration, see Figure 1 of Condran et a/. 1991). 

Age overstatement that is confined to deaths will create 
a pattern of ratios that is above 1.00 and rises with age; 
the denominator is too low (its negative component is too 
large) and the proportionate deficit grows with age. 

Introducing the same pattern of age overstatement into 
deaths and population figures also creates ratios that even- 
tually rise with age. This important result is robust to the 
extent of error introduced (Condran et al. 1991). It reflects 
the fact that age distributions taper off more and more 
rapidly as age advances, so that the same percentage of 
persons who overstate their true age will introduce larger 
percentage errors in the reported age distributions at the 
very advanced ages. That is, N,(2) has a larger inflation 
factor than N,_j9(1). In this case, some inflation in 
N,—19(1) is offset in its effects on the denominator by an 
inflation in D. 


6. RESULTS 


Intercensal cohort analysis was carried out for the four 
sex-race groups in the United States in the 1970-1980 and 
1980-1990 periods. Figures | and 2 present the calculated 
ratios of the observed to expected population at selected 
ages by race, sex, and intercensal period. 

In all race-period combinations, the age pattern of ratios 
is virtually the same for females and males. In all cases, 
the degree of inconsistency increases with age, although 
any systematic and significant departure from 1.00 is post- 
poned until age 95 and beyond for whites in 1980-1990. 
There is clearly a discontinuity in many of these series at 
age 100, reflecting the idiosyncrasies of age reporting and 
Census Bureau adjustment procedures among centenarians. 
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Figure 1A. Intercensal Ratios of Observed to Expected 
Population: Whites, 1970-1980. 
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Figure 1B. Intercensal Ratios of Observed to Expected 
Population: Whites, 1980-1990. 
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Figure 2A. Intercensal Ratios of Observed to Expected 
Population: Blacks, 1970-1980. 
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Figure 2B. Intercensal Ratios of Observed to Expected 
Population: Blacks, 1980-1990. 
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6.1 Results for Whites 
6.1.1 Intercensal Period: 1970-1980 


As shown in Figure 1A, the white pattern in 1970-1980 
is generally above unity and rising with age (up to age 100). 
This pattern is consistent with several forms of data error, 
the two most plausible patterns of which are: 


1) Undercount in the 1970 census, relative to both the 
1980 census and the death registration. 


2) Roughly equal probabilities of age overstatement in 
deaths and in both censuses. 


’ We believe that the former explanation is more likely 
to be correct. If the pattern of ratios resulted from similar 
tendencies for age misstatement in deaths and censuses, 
one would expect that pattern to continue into the 1980-1990 
decade, particularly since the 1980 census is involved in 
both comparisons. And one would not expect cultural 
predispositions to misstate age to disappear suddenly. But 
the 1980-1990 pattern of ratios for whites (Figure 1B) 
shows remarkable consistency, far better than that in most 
European countries and equivalent to the pattern of ratios 
found in Sweden and the Netherlands, countries with 
highly efficient population registers (Condran et al. 1991). 
The consistency during 1980-1990 is also much greater 
than that in other English-speaking countries: England 
and Wales, Canada, Australia and New Zealand. 

A second reason for accepting the first explanation is 
that the Census Bureau has concluded that the 1980 census 
is more complete than the 1970 census (U.S. Bureau of the 
Census 1988; Robinson ef a/. 1993). This conclusion is 
partially based on demographic analysis and hence is not 
entirely independent of the kind of evidence that we are 
reviewing. However, their demographic analysis is weighted 
heavily towards ages that are younger than those consid- 
ered here. Furthermore, the conclusion that census cover- 
age improved is also supported by their post-enumeration 
program in which individuals in the census are matched 
against other data systems. 


6.1.2 Intercensal Period: 1980-1990 


As noted earlier, the 1980-1990 pattern of ratios for 
whites (and particularly for white females) is highly con- 
sistent, far better than in most European countries. Our 
investigation seemingly lends support to Vaupel’s (1993) 
contention that the white population of the United States 
may have lower death rates above age 80 than any other 
industrialized country. But caution is in order. While our 
methods clearly highlight the consistency between the 
censuses and the death registraticn in 1980-1990, consis- 
tency is not equivalent to accuracy. Condran et al. (1991) 
demonstrate one situation in which a pattern of age mis- 
reporting can result in a ratio series at exactly 1.00 at all 
ages. Furthermore, the intercensal methodology fails to 
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capture deliberate misreporting of age by individuals that 
is consistent over time. As noted by Horiuchi (1993), an 
initial overstatement of age - e.g., to allow entrance into 
school or the labor force at a younger age, to avoid being 
drafted near the upper limit of drafting age, or to receive 
Social Security, Medicare, or pension payments earlier - 
may be followed by consistent, intentional overstatement 
of age. The possibility of such overstatement of age cannot 
be discounted although we are unable to measure it directly. 


6.2 Results for Blacks 


In contrast, the pattern of ratios for African-Americans 
is far more regular over time (see Figure 2A and 2B). The 
ratios begin falling around age 70 for both sexes in both 
periods and continue falling through higher ages (until age 
100 in 1970-1980). Before age 70, ratios are typically well 
above unity in 1970-1980, and slightly above 1.00 for 
African-American males during 1980-1990. 

The fact that ratios are generally higher for African- 
Americans at a particular age in 1970-1980 than in 1980-1990 
is consistent with a relative undercount in the 1970 census. 
As we noted earlier, such an undercount is also likely to 
have occurred among whites. The undercount, however, 
is insufficient to explain the persistent pattern of falling 
ratios above age 70 in both periods. The declining ratio 
series for African-Americans is consistent with two prin- 
cipal explanations: 


1) Deaths are underregistered for the African-American 
population relative to completeness of census coverage. 


2) Age overstatement is greater in censuses than in death 
registration. 


Coale and Kisker (1990) lean toward the former expla- 
nation. They note that populations reconstructed from 
deaths using variable-r procedures (Preston and Coale 
1982) are too small relative to census counts in 1980 above 
age 65, suggesting relative underregistration of deaths. 
They also note that fewer African-American deaths are 
recorded at advanced ages in vital registration than in 
Medicare records. 

However, both observations are also consistent with 
ages being overstated in censuses (and Medicare) relative 
to death registration. That such a pattern exists is strongly 
supported by a direct match of death certificates in 1960 
to records for the same individuals in the 1960 census of 
population (NCHS 1968; Hambright 1969). For either 
males or females, the total number of deaths above age 50 
when deaths are classified according to census age are 
within 1% of the total number of deaths when classified 
according to death certificate age. However, at ages 65+, 
‘‘census age’’ deaths are 15.4% greater than ‘‘death 
certificate age’’ deaths for females and 7.1% greater for 
males. At age 75 + , the disparities are 23.3% and 17.8%, 
respectively, and at age 85+, 39.2% and 17.6%. 
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These large discrepancies in age reporting between 
censuses and deaths are capable of accounting for the 
declining pattern of ratios above age 70 that is demonstrated 
in Figure 2. Elo and Preston (1994) calculate the R, values 
for African-Americans between 1950-1960 and 1960-1970, 
periods that bound the 1960 census-death certificate 
match. They show that, if ages at death are ‘‘corrected’’ 
to make them consistent with the age reporting in the 
censuses, the pattern of declining ratios is eliminated. 

Reasons why African-American ages are overstated in 
censuses relative to deaths are not obvious. The pattern 
does not appear until the 1940 census, the first census after 
Social Security legislation was passed. At that census, a 
large surplus of African-American persons aged 65-69 and 
70-74 appears, and a deficit of persons aged 50-64 (Elo 
and Preston 1994). As noted by Wolfenden (1954:56), 
‘‘the disturbances were so marked in the data for Negroes 
that special preliminary redistributions of those populations 
(and deaths) between 55 and 69 were made in the prepara- 
tion of the [U.S.] life tables.’’ This surplus also appears, 
although in increasingly attenuated form, in more recent 
censuses (as shown in Figure 2). Whatever its source, we 
believe that the principal explanation of the large incon- 
sistencies between censuses and death registration for the 
African-American population is a pattern of age over- 
statement in censuses relative to death registration. Such 
a pattern implies that recorded death rates above age 65 
for African-Americans are likely to be seriously under- 
estimated. A cross-over between black and white death 
rates may indeed occur at advanced ages, but basing such 
a conclusion on U.S. census and vital registration data is 
treacherous. These data are simply too inconsistent with 
one another to allow death rates at advanced ages to be 
estimated with any confidence. 


7. CONCLUSION 


Major uncertainties about the quality of elderly popula- 
tion and death enumerations in the United States result 
from coverage and content errors in the censuses and the 
death registration system. This study evaluates the consis- 
tency of reported data between the two sources for the 
white and the African-American populations. The focus 
is on the older population (aged 60 and above), where 
mortality trends have the greatest impact on social pro- 
grams and where data are most problematic. Using inter- 
censal cohort analysis, age-specific inconsistencies between 
the sources are identified for two periods, 1970-1980 and 
1980-1990. 

In order to evaluate what combinations of coverage 
completeness and age misreporting patterns would produce 
the empirical results, a series of simulations were carried 
out. The U.S. data inconsistencies are examined in light 
of both the simulation results and evidence in the literature 


regarding the nature of coverage and content errors in the 
data sources. 

Data for whites in the 1980-1990 intercensal period were 
found to be remarkably consistent. Data quality up to age 
95 approaches that of Sweden and the Netherlands, coun- 
tries which maintain highly efficient population registers. 
Less consistency was observed for whites during the 
1970-1980 decade. The most likely explanation for this 
pattern of inconsistencies is the relative net undercount in 
the 1970 census combined with more complete death 
statistics. Consequently, mortality estimates at older ages 
that combine numerators from the death registration with 
denominators from the 1970 census are likely to overstate 
mortality. 

A different pattern is observed in the African-American 
data. Above age 70, the enumerated population falls 
increasingly below the expected population in both 1980 
and 1990. It appears that the major reason for this pattern 
is that ages are overstated in censuses relative to death 
registration. Such a pattern implies that recorded death 
rates at older ages for African-Americans are likely to be 
seriously underestimated. A mortality crossover between 
black and white death rates may occur at advanced ages, 
but basing such a conclusion on census and vital regis- 
tration data is hazardous. 
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APPENDIX A 
Source: Shrestha (1993) 


Three major sources of data were utilized in this 
research: (1) census enumerations for 1970, 1980, and 
1990; (2) official death registration data; and (3) net 
immigration statistics. Sources of the data and adjust- 
ments made will be described. 


1.A The 1970 Census 


Official tabulations of the 1970 population by basic 
demographic characteristics are presented in Series B - 
U.S. Summary of the 1970 Census (U.S. Bureau of the 
Census 1972). The official enumerations are known to 
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contain a number of major inaccuracies which could bias 
our investigation of the enumerated old-age population in 
the United States. The first is a conspicuous overcount of 
the centenarian population. Whereas 106,000 persons were 
enumerated in the open-ended category, indirect demo- 
graphic analysis estimates the correct count to be in the 
range of 3,000 to 8,000 (Siegel 1974; Siegel and Passel 
1976). The overcount appears to have been the result of 
misunderstanding of the census form rather than systematic 
age misreporting into the centenarian population. The 
second problem is the result of misclassification of the 
population by race in the complete-count tabulations, 
affecting 21,000 individuals aged 65 and above. And 
finally, the official count omitted over 23,000 individuals 
(of all ages) whose records were discovered after the initial 
tabulations were published. 

Because of the inherent errors in the official tabula- 
tions, we utilize unpublished adjusted tabulations obtained 
from the U.S. Bureau of the Census. The modified 
statistics include corrections for the three previously men- 
tioned problems. The data are presented by race (white, 
black), sex, and age (single years of age 0-94 and grouped 
data 95-99, 100+). To distribute the grouped data from 
age group 95-99 to single years of age, we used the sex-and 
race-specific average age distribution from the 1960 and 
1980 censuses for whites, and from the 1950 and 1980 
censuses for blacks (data by single years of age is not 
available in this age range for blacks in the 1960 census). 


1.B The 1980 Census 


Originally published census tabulations for 1980 were 
presented in Series B - U.S. Summary of the 1980 Census 
(U.S. Bureau of the Census 1983). In the 1980 census, 
however, a large number (about 6.8 million) of persons 
enumerated chose to write-in a response to the race question 
as opposed to selecting one of the specified all-inclusive 
race categories. Since only the 1980 census contained a 
residual race category, the official enumeration was not 
directly comparable with other data sources (vital registra- 
tion, earlier censuses, efc.). The Census Bureau produced 
a modified file which conforms to the historical categories 
of the racial groupings (U.S. Bureau of the Census 1984b). 
The modification procedure involved macro-level reassign- 
ment of race based on detailed cross-tabulation of race and 
Hispanic-origin from the sample and complete-count 
census data. The specifics of the Census Bureau modifi- 
cation follow. 

For the 219.8 million individuals who chose one of the 
14 specified categories, no adjustment was made. Two cat- 
egories of individuals, totalling 6.7 million, with write-in 
responses were identified: persons of Hispanic-origin 
(5.8 million) and persons not of Hispanic-origin (0.9 million). 
Separate adjustment procedures for the two groups were 
developed. 
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Those of Hispanic-origin were distributed only to the 
white or black categories (and not to American Indian or 
Asian/Pacific Islander categories). All persons of Mexican 
origin were reassigned as white. Persons of Puerto Rican, 
Cuban, and other Spanish origin were assigned to both 
white and black modified race groups on the basis of the 
distribution of the same Hispanic-origin individuals who 
originally specified either a white or black race on the 
census returns. The calculations were carried out within 
age-sex-county cells. 

Those not of Hispanic-origin were reassigned to all 
three modified race groups (white, black, other) on the 
basis of state-specific proportions which are applied to all 
age-sex-county cells within the state. The proportions are 
based on sample data from the 1980 census. For a more 
detailed discussion of the modifications, see U.S. Bureau 
of the Census 1984b. 

The modified tabulations are presented by race, sex, 
and single years of age (0-99; 100+). We utilize the race- 
modified statistics in this research, justified by the sheer 
magnitude of persons transferred from the residual race 
category to the white or black categories. 


1.C The 1990 Census 


Published tabulations of the 1990 Census continue 
to be released by the U.S. Bureau of the Census. The 
published statistics, however, contain a number of problems 
that make comparability with earlier censuses and other 
sources of data difficult. Three problems are apparent: 
racial classification of 9.3 million individuals in a residual 
non-specified racial category, inconsistencies in the 
reporting of age, and a change in allocation procedures 
for the 1990 census in assigning age to persons with missing 
data on the characteristic. 

A modified 1990 census file, referred to as the MARS 
(Modified Age and Race Statistics) was produced at the 
Census Bureau to adjust for the first two problems (Word 
and Spencer 1991). Modification of the 1990 census was 
conducted at the micro-level. Hot-deck imputation proce- 
dures were utilized to assign a specific race to persons who 
reported themselves in the ‘‘other, not specified’’ racial 
category. The method is executed on the individual records 
of the 100% edited detail file from the 1990 census 
(Robinson, Word and Spencer 1991). 

We again utilize the modified statistics, which are 
tabulated by race, sex, and single years of age, in this 
research. The decision to use the modified statistics in both 
1980 and 1990 was not clear-cut. See Shrestha 1993 fora 
more detailed discussion. 


2. The Death Registration System 


National-level annual death statistics from the National 
Center for Health Statistics (NCHS) are utilized in this 
research. The data for 1970 through 1988 are extracted 
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from NCHS data tapes obtained from ICPSR (NCHS 
1970-1988). The data are provided by race (black, white), 
sex, and single years of age (0-124; 125+). Since the data 
tapes for calendar year 1989 and for the first three months 
of 1990 had yet to be released, we developed a procedure 
to estimate the distribution. Final mortality statistics for 
1989 by race and sex were released in published form by 
NCHS (1992). The grouped age data was distributed to 
single years of age based on the 1988 death distribution 
within the grouped age category. Distribution to month 
of death was based on monthly vital statistics reports 
(NCHS 1989). Estimates of the death distribution in 1990 
are based on monthly advance reports of mortality from 
NCHS (1990). The preliminary numbers were distributed 
to single years of age again using the 1988 distribution 
within the grouped age category. 

As noted in the text, we adjusted the available data to 
correct for two problems. First, the intercensal period 
covers the interval from April 1 to March 31, whereas the 
death registration data refer to calendar years. Second, 
both sets of data are reported by age at last birthday rather 
than by year of birth. Because the census is on April 1, the 
latter is preferred because it identifies the birth cohort for 
use in cohort analysis. To adjust for these two problems, 
we assume that the three dimensional surface of the 
number of deaths in age and time is level over the interval. 
We do not adjust for underregistration nor for misreporting 
of characteristics in the death statistics. 


3. Net Immigration Statistics 


We utilize unpublished net immigration statistics 
obtained from the U.S. Bureau of the Census. The tabula- 
tions are categorized in the form of ‘‘components of 
change’’ for each of the two decades. 

Age-, race-, and sex-specific net immigration was calcu- 
lated on a cohort basis by use of the following equation: 


Net immigration = Legal Alien Immigration + Refugees 
and Parolees + Net Civilian Citizens Immigration 
+ Net Puerto Rican Immigration + Net Foreign 
Students Immigration + Net Movement of U.S. 
Armed Forces Overseas — Legal Emigration. 


Given the lack of sufficient detail in the raw data provided 
by the U.S. Census Bureau, a number of adjustments were 
required. First, the data had been provided with an early 
terminal age group (age 75 and above at the beginning of 
the decade). To distribute to five-year age groups (75-79, 
..., 95-99, 100+), we assumed that the age-, race-, and 
sex-specific net immigration rate for ages 75 + remained 
constant in the open-ended interval beginning at age 75. 
This admittedly crude estimate is adequate because of 
small numbers of net immigrants in this age group. 
Second, to convert the five-year data into single years of 
age, we used Sprague multipliers or osculatory interpolation 
(Sprague 1880-81). 
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An Assessment of the Use of Hand-Held Computers During 
Demographic Surveys in Developing Countries 
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ABSTRACT 


Although large scale surveys conducted in developing countries can provide an invaluable snapshot of the health 
situation in a community, results produced rarely reflect the current reality as they are often released several months 
or years after data collection. The time lag can be partially attributed to delays in entering, coding and cleaning 
data after it is collected in the field. Recent advances in computer technology have provided a means of directly 
recording data onto a hand-held computer. Errors are reduced because in-built checks triggered as the questionnaire 
is administered reject illogical or inconsistent entries. This paper reports the use of one such computer-assisted 
interviewing tool in the collection of demographic data in Kenya. Although initial costs of establishing computer- 
assisted interviewing are high, the benefits are clear: errors that can creep into data collected by experienced field 
staff can be reduced to negligible levels. In situations where speed is essential, a large number of staff are involved, 
or a pre-coded questionnaire is used to collect data routinely over a long period, computer-assisted interviewing 
could prove a means of saving costs in the long term, as well as producing a dramatic improvement in data quality 
in the immediate term. 


KEY WORDS: Hand-held computers; Demographic surveys; Psion. 


1. INTRODUCTION Table 1 


Summary of Chronology of 12 African WFS Surveys 
Large scale surveys involving tens of thousands of (Source: WES 1986) 


respondents, such as national censuses, demographic or 


; ; : Number 
health surveys, are routinely conducted in developing ar 
countries. Their intention is to provide rapid, up-to-date Number te Datene Months 
information on population and health issues for evalua- Country ' of Survey hire oe 
tion and planning purposes. Their wide scope necessitates ee Started Report eee 
numerous personnel comprising trainers, interviewers, Report 
supervisors, data entry staff and data managers. Examples Date 
of such questionnaire-based surveys include the World 
Fertility Survey (WFS 1986) and national Demographic neni 4,018 12/1981 06/1984 30 
and Health Surveys (DHS Kenya 1989). Published dates 
for the commencement of the WFS surveys in 12 African Cameroon 8,219 01/1978 = 04/1983 63 
countries and the dates the first country reports were Ghana 6,125 02/1979 06/1983 52 
produced (Table 1) illustrate the time required before data Ivory Coast 6,270 08/1980 12/1984 52 
was available for planners to act upon (WFS 1986). On 
average it took 45.6 months before the final report was env Be ommmmcn aT) CaAlee0 ss 
released. Survey logistics in developing countries undoubt- Lesotho 3,603 08/1977 12/1981 52 
edly contribute to delays in provision of completed data; Mauritania 3,500 01/1981 06/1984 4] 
so do the mechanics of data processing. The recent Demo- Neeheen 5,800 04/1980 05/1984 49 
graphic and Health Survey conducted in Kenya required 
five data entry clerks, two data entry supervisors and a Nigeria 9,727 LO/I981' 09/1984 35 
control clerk to process 8,343 household interviews; data Senegal 3,985 05/1978 07/1981 38 
collection began in February 1989 and the first draft of the Sudan (North) 3,115 12/1978 04/1982 40 
final report was ready for circulation seven months later ae 4,123 05/1978 06/1983 él 


(DHS Kenya 1989). 


!'D. Forster, Department of Tropical Medicine, University of Oxford, John Radcliffe Hospital, Headington OX3 9DU, England; R.W. Snow, 
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Surveys of this size involve multiple levels of checking 
and coding of data collected in the field providing another 
source of delay. As speed underpins rapid health evaluation 
(Anker 1991; Vlassoff and Tanner 1992), reducing the time 
at this check and code stage is a major advantage to the 
survey process. Advances in computer hardware have led 
to the development of microcomputers suitable for use in 
field situations. Together with improved software designed 
for questionnaire specification and administration, 
computer-assisted interviewing is now a viable option. 
National statistics offices in industrialised countries have 
evaluated the use of this technique, and some now use 
them on a regular basis (Nicholls and Groves 1986; Lyberg 
1985; Denteneer et al. 1987; Bench ef al. 1994). The 
advantages of these systems are that it reduces recording 
errors by simplifying skip modules and refusing inappro- 
priate, illogical or inconsistent entries. Furthermore, large 
numbers of interviews can be stored and simply down- 
loaded to a central computer at the end of every interview 
session, circumventing the need for data entry clerks. 

There is surprising reluctance to adopt this technology 
in developing countries despite its apparent advantages. 
There are several possible reasons for this. Firstly, the 
initial costs may seem daunting and the application deemied 
inappropriate in countries with scarce resources. Secondly, 
there have been few attempts to validate their use under 
field conditions providing little quantifiable evidence 
of their limitations or advantages over traditional data 
collection techniques (Reitmaier 1985; Ferry and Cantrelle 
1988; Forster et al. 1991). This paper presents the results 
of a comparative study of two methods of field data 
collection and processing conducted during a demographic 
survey on the Kenyan Coast. 


2. THE ADULT MORTALITY SURVEY 


The study was carried out as part of ongoing demo- 
graphic and epidemiological studies of 60,000 people living 
on the Kenyan coast. The study population and survey 
methods employed to monitor demographic events has 
been described elsewhere (Snow ef al. 1994). In brief, 
following an initial census of the population all vital events 
are monitored by means of 6-weekly house-to-house visits 
and bi-annual re-censuses of the entire population. During 
a re-enumeration of the population in November 1993, a 
survey was undertaken to estimate adult mortality using 
indirect demographic methods (Timaeus 1991). All women 
aged between 25 and 44 years were interviewed using the 
structured questionnaire as shown in Figure 1. The format 
used precoded closed questions, with logical skips and a 
consistency check. 

Twenty-four field staff, all secondary school leavers, 
were involved in the survey. All were familiar with survey 
and census procedures, having had previous formal training 


in field survey techniques and between 1 and 5 years of 
field experience. Two days was spent on additional training 
on the administration of the adult mortality questionnaire. 
During the survey field staff were divided into two teams, 
each supervised by a senior fieldworker. Questionnaires 
completed at the end of each day were checked by field 
supervisors then passed to the computer staff for data entry. 
This was done using a screen design reflecting the structure 
of the paper questionnaire in FoxPro (version 2.0). The 
same data was independently entered by two data entry 
clerks and the two completed files compared to identify 
entry errors, which were subsequently corrected. The com- 
pleted file was then subjected to logical, range and consis- 
tency checks; these included for example, the identification 
of missing data, incorrect coding (i.e., not using ‘‘Y’’ or 
‘*N’’), dates inconsistent with the ages of the women and 
the date of the survey (questions 5 and 6 in Figure 1) and 
checks that the sums of questions 7, 9, 11 and 13 are 
consistent with question 15 as shown in Figure 1. 


3. COMPUTER DATA COLLECTION TEST 


3.1 Computer Hardware and Software 


An earlier version of questionnaire-based software was 
developed for the Psion Organiser II (Forster et al. 1991). 
This model had a limited screen size, 16 characters by 
2 lines, but had a fully operational keyboard. The Psion 
Series 3, used during the present study, offers new possi- 
bilities: the screen is much larger, with 40 characters by 
8 lines, and integrated graphical capabilities. The machine 
remains small (165mm by 85mm by 22mm), and weighs 
265g including 2 AA sized batteries. The storage devices 
can store up to 1 megabyte. The keyboard is a 58 key, 
QWERTY layout. Communications between the Psion 
Series 3 and a PC entails a simple copy operation between 
the two storage media. 

The software was developed using Psion’s in-built 
programming language, OPL. The paper questionnaire is 
represented in a structured format ina text file, according 
to a prescribed format. The questionnaire definition 
includes a mixture of questions and commands such as 
skips and range checks. The internal range checks included 
those developed for the inconsistency checks for the data 
entered using FoxPro described above. Data entered on 
the Psion is stored in a separate file, one line for each 
interview. 

To specify a question correctly it must include a ques- 
tion number, the question text and the answer type, which 
can be a list option, a character input or a number. The 
definition should also indicate what position in the line the 
corresponding data entry should be stored and how long 
the entry is. Numeric answers can also include a prespecified 
number of decimal points. A range of acceptable inputs 
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Figure 1. The Adult Mortality Questionnaire 


Questionnaire on the survival of relatives 
(For all women aged 25-44 years) 


Names 
Date ID - - 


| would like to ask you some questions about your natural parents and about your brothers and sisters who have 
the same mother as you. 


Is your mother alive? (1 = yes, 2 = no) 

Is your father alive? (1 = yes, 2 = no) 

INTERVIEWER: If both parents alive (Q1 and Q2 = 1), go to Q7. 
Have you ever given birth? (1 = yes, 2 = no) 

INTERVIEWER: If she has never given birth (Q3 = 2), go to Q6. 


Was (MENTION ALL PARENTS NOT ALIVE NOW) alive 
at the time that you gave birth to your first child? 


Woman's mother 
Woman’s father 

In what year was your first child born? 

In what year (MENTION ALL PARENTS NOT ALIVE NOW) die? 
Woman's mother 
Woman’s father 

How many living sisters, born to your mother, 

do you have? (ALIVE NOW) 

INTERVIEWER: If no living sisters (Q7 = O), go to QQ. 

How many of these living sisters are less than 15 years old? 

How many of your sisters, born to your mother, have died? 

INTERVIEWER: If no dead sisters (Q9 = 0), go to Q117. 


How many of these dead sisters died before age 15 years? 


How many living brothers, born to your mother, 
do you have? (ALIVE NOW) 
INTERVIEWER: If no living brothers (Q11 = OQ), go to Q73. 
How many of these living brothers are less than 15 years old? 
How many of your brothers born to your mother, have died? 
INTERVIEWER: If no dead brothers (Q13 = 0), go to Q15. 
How many of these dead brothers died before age 15 years? 
INTERVIEWER: Sum Q7, 9,11 and 13: Q7 

Q9g 

Q11 

Q13 = 
| want to make sure that | have this right. Apart from you, your mother had 
children altogether? Is that correct? 
INTERVIEWER: In the case of any inconsistency, probe and correct Q7 to Q14 if necessary. 


INTERVIEWER: Please thank the woman for her co-operation. 


Fieldworker code 
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is an optional specification for numeric or character 
answers and will include a minimum, a maximum or both. 
List options can be used to specify codes and their values. 

Command actions can be embedded in question texts, 
so that they are evaluated at the time of questionnaire 
administration. For example, the final cross-check question 
in Figure 1 requires an addition. The syntax allows this 
instruction to be included within the main body of the 
question text. Other commands can contain instructions 
on skipping a question, conducting a cross-check between 
answers or for moving to a different question. 

Thus the software uses a flexible way of defining a 
questionnaire, which is generally applicable. It incorporates 
manipulation of entered information, integrating arithmetic 
functions into questions or command lines. The next step 
forward would be to design an interface for questionnaire 
specification which removes the burden of constructing a 
syntactically correct questionnaire definition. The soft- 
ware is available from the authors. 


3.2 Test Design 


An additional day’s training on the use of the Psion was 
provided for the two team leaders. This involved an expla- 
nation of the hardware and software, as well as practice 
sessions in the field. Both supervisors had had no previous 
computing experience. Both conducted interviews using 
either the Psion or paper questionnaires on alternate days, 
and these formed the basis of the comparison between the 
methods. 

Errors made by all the 22 fieldworkers using the paper 
questionnaire were counted and tabulated to estimate the 
background error rate using this method of data collection. 
Times taken to check the forms once they had been brought 
back from the field, and for the data to be entered, verified 
and corrected following range and consistency checks were 
recorded throughout the survey. Similar timing assessments 
were made for the Psion data collection procedures. 


4. TEST RESULTS 


4.1 Time 


The average length of interviews conducted on paper 
was 5.1 minutes, and for those on computer 5.0 minutes, 
demonstrating no difference between the two methods 
(Table 2; note only 215/234 interviews were timed). The 
length of interviews varied considerably from 1 to 18 
minutes and increased time related not simply to the 
number of skips made on each interview, but whether the 
respondent gave clear, non-contradictory answers. 

Team supervisors required between 2-3 hours per day 
to check each teams questionnaires. The average time 
taken to enter 500 records (approximate number of inter- 
views completed per week) was 3 hours 40 minutes per data 


Table 2 


Comparison Between Paper Questionnaire and Psion 
Series 3 Data Collection Methods 


Length of Interview in Minutes 


Minimum Maximum Average Average Average 


Overall Overall Overall Leader A Leader B 
Paper 1 16 Sell (215) Raed 28) eA LOL) 
Computer 1 18 5.0 (363) 5.5 (190) 4.5 (173) 


* Number of timed interviews stated in brackets. 


entry clerk; double entry required 7 hours 20 minutes 
(Table 3). Verification required an additional 2 hours 
23 minutes for the same number of questionnaires. The 
completed files were edited twice to reflect corrected 
errors and then verified; this took on average two hours 
30 minutes for 500 records. 


Table 3 


Times for Data Processing 
500 Questionnaires 


Activity Average time 


Data checking 4 hours 8 minutes 


First data entry 3 hours 40 minutes 


Second data entry 3 hours 40 minutes 


Verification 2 hours 23 minutes 

Editing 2 hours 33 minutes 

Total time 18 hours 24 minutes 
4.2 Errors 


The errors made were divided into two periods, to 
assess the effect of familiarity over time. Excluding the 
two team leaders, the remaining 22 fieldworkers made 
1,704 errors on 1,427 questionnaires in the first period, 
and 1,049 errors on 1,158 questionnaires in the second 
period. Thus the average error rate per questionnaire in 
the first two weeks was 1.19 and in the third and fourth 
weeks was 0.90. In addition, over the entire period 37 ques- 
tionnaires (1.2% of all interviews) had to be sent back to 
the field to be redone, as the errors found were not recon- 
cilable in the office. These questionnaires had between 1 
and 6 errors to be corrected, with a total of 61 errors. 
The highest number of errors were made on question 5 
(17 errors) and question 6b (15 errors). Error rates per 
question are shown in Table4. Fourteen out of the 
22 fieldworkers redid at least one questionnaire. One 
fieldworker was required to redo 8 questionnaires. 
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Table 4 


Type of Errors Made by 22 Fieldworkers 
Using Paper Questionnaires 
(for question specification see Figure 1) 


Period 1 
(first fortnight) 


Period 2 
(second fortnight) 


Identification 163 48 
Question | 6 1 
Question 2 8 2 
Question 3 125 92 
Question 4a 201 138 
Question 4b 1531 93 
Question 5 105 61 
Question 6a 94 57 
Question 6b 65 41 
Question 7 14 0 
Question 8 109 63 
Question 9 51 10 
Question 10 178 134 
Question 11 13 1 
Question 12 108 71 
Question 13 55 3 
Question 14 204 149 
Question 15 19 76 
Fieldworker code 37 9 
Total errors 1,704 1,049 
Total questionnaires 1,427 1,158 


Errors were detected either manually by final checking 
by one of the investigators (Forster) or through the range 
and consistency checks performed in FoxPro on the entered 
data. Field team leader A made 8 errors on 144 question- 
naires (0.06 errors per questionnaire), and team leader B 
made 18 errors on 90 questionnaires (0.20 errors per ques- 
tionnaire). Most of these errors occurred in question 10 
(4 errors) and question 15 (5 errors). The only errors found 
from the computer data were errors of respondent iden- 
tification. There were 7 of these, 2 by leader A and 5 by 
leader B, giving errors of 0.01 and 0.03 per questionnaire 
respectively. Such errors could have been circumvented by 
pre-loading the Psion with a call list of respondents to 
interview. 


4.3 Cost 


The differential costs of a survey of this size using 
Psion-based and paper-based methods are given in Table 5. 
The Psion prices quoted are the recommended retail prices. 
Intense competition between retailers means that purchase 
prices could be up to 20% lower than those quoted here. 
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Prices of hardware products are also decreasing. Current 
prices indicate that the one off cost of a Psion-based 
system can be recouped after 12-15 similar paper-based 
surveys of approximately 7,000 respondents. 


Table 5 


A Comparative Study of Computer-based and Paper-based 
Survey Methods (UK £ Sterling) 


Equipment required Cost 
20 Psion Series 3 2,539.00 
20 1 MB storage devices 2,039.00 
Computed- : on ; 
aad 1 serial communications link 59.45 
survey 80 rechargeable batteries 146.20 
1 battery recharger 15.95 
Total cost 4,799.59 
14 reams of paper for 7,000 interviews 42.00 
Duplicating costs for 7,000 ques- 
tionnaires (double-sided) 70.00 
Paper- 20 pens, erasers and correcting fluid 27.40 
based 20 clipboards 100.00 
survey 2 data entry clerks (two weeks) 70.00 
2 supervisors* (one month plus 
overtime) 85.00 
Total cost 394.40 


* Necessary for the manual checking of forms as they come in from 
the field each day. 


5. DISCUSSION 


The lowest error rates using a traditional paper ques- 
tionnaire by senior field workers with five years of data 
collection experience was on average 0.11 errors per ques- 
tionnaire with 17 fields. This was reduced to negligible 
levels using the questionnaire software developed for a 
Psion Series 3 hand-held computer. This technique elimi- 
nated most of the errors made by fieldworkers in the 
routing of the questionnaire (Table 4) by using pre-defined 
skip modules, thus reducing the error rate by at least 90%. 
With the additional implementation of a call list in the 
software, the rate of respondent identification errors 
would be even lower. 

The field supervisors were keen to use the computer, 
mastering the unfamiliar QWERTY keyboard, and learnt 
operating procedures quickly enough to take to the field 
without supervision after two days. Although no formal 
investigations were under taken to gauge and quantify 
interviewees’ reactions to the Psion there were surprisingly 
few comments about the computer and no interview 
refusals. 

Data processing involved two data entry clerks using 
two IBM machines full time for 92 hours to complete the 
data entry process for the entire survey. A data manager 
was on hand to offer assistance where necessary and design 
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the data entry format. The setting of the present study was 
such that both data entry clerks were familiar with data 
entry procedures and the available hardware and software. 
In situations where this is not the case, closer supervision 
and involvement by a data manager would be necessary, 
thus incurring an additional cost. A Psion data collection 
system would require much less of a data manager’s time 
to down-load each day’s data, thereby reducing this com- 
ponent of staff costs. Never-the-less, the initial cost of the 
Psion Series 3 may be prohibitively expensive when com- 
pared to the costs of paper and duplication of question- 
naires if it was not envisaged that they form part of future 
data collection activities. 

QUESTOR (Ferry and Cantrelle 1988) offers a suitable 
software environment for computer-assisted interviewing. 
However, the hardware required is a portable PC, several 
times the costs of a hand-held Psion. Our experience 
demonstrates that it will be worth pursuing the develop- 
ment of an appropriate package using this compact PC 
compatible technology as a more practical alternative in 
the field, being easier to handle, more robust and with 
reduced power consumption. 

There is a trade off between error rates, time and cost 
of a survey. The use of computer-assisted interviewing 
software can reduce both the error rates and the length of 
time for data preparation considerably. Such a collection 
system should reduce the unacceptable delays in first 
presentation of data experienced during surveys such as 
the World Fertility Survey (Table 1). The context of the 
present comparative study differs from many large scale 
demographic surveys where recruited fieldstaff are un- 
familiar with questionnaire procedures. We feel that the 
results presented here therefore represent a minimum 
improvement that could be expected in data quality. The 
initial cost of setting up such a survey mechanism may be 
daunting, but will be proportionally less for repeated 
surveys, or in institutions conducting a variety of different 
surveys over time. 
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Statistical Process Control of Sampling Frames 


A.W. SPISAK! 


ABSTRACT 


Statistical process control can be used as a quality tool to assure the accuracy of sampling frames that are constructed 
periodically. Sampling frame sizes are plotted in a control chart to detect special causes of variation. Procedures 
to identify the appropriate time series (ARIMA) model for serially correlated observations are described. Applications 
of time series analysis to the construction of control charts are discussed. Data from the United States Department 
of Labor’s Unemployment Insurance Benefits Quality Control Program is used to illustrate the technique. 


KEY WORDS: Autocorrelation; ARIMA models; Control charts; Quality assurance. 


1. INTRODUCTION 


The integrity of the sampling frame is of paramount 
importance in survey research. Frame imperfections include 
missing elements (incomplete frame), element clusters 
(more than one element in a single listing), blank or foreign 
elements, and duplicate listings. These imperfections can 
cause several difficulties by contributing to nonsampling 
error, reducing the number of sample cases from sub- 
classes of the population, and requiring the use of complex 
weights to estimate population characteristics. Techniques 
to minimize frame problems or reduce their impact on the 
survey are discussed in detail in most textbooks on statis- 
tical surveys. 

This article focuses on the statistical process control of 
sampling frames which are constructed periodically (daily, 
weekly, or monthly, for example) and which consist of 
elements that are generated by a continuous process. 
Because of the variation inherent to any dynamic process, 
the sizes of the sampling frames will vary. How do we 
know that the changes in the sizes of the sampling frames 
reflect the random variation of the process and not errors 
in the construction of the frames? Statistical process 
control allows survey managers to distinguish between the 
variation inherent in the process (common causes) and 
variation which signals a possible problem with frame 
construction (special causes). 


2. PROCESS VARIATION AND STATISTICAL 
PROCESS CONTROL 


Over the last several years managers in the manufac- 
turing, service, and public sectors of the economy increas- 
ingly have adopted the quality philosophies developed by 
W. Edwards Deming, J.M. Juran, Philip B. Crosby, Kaoru 
Ishikawa, and others. Quality management comprises an 


array of tools and techniques, including the use of con- 
trol charts to determine if a process is in statistical control. 
According to Deming (1982), statistical control is achieved 
by eliminating special causes of variation, leaving only the 
random variation of a stable process. The behavior of a 
process that is in statistical control is predictable. 

The distinction between common and special causes of 
variation is a key principle of statistical process control. 
Deming (1982) credits Dr. Walter A. Shewhart, who devel- 
oped many of the principles of statistical process control 
in the 1920s and 1930s, with originating the concept of 
special or assignable causes. Special causes are usually 
attributable to one part of the process, such as a worker, 
machine, or office. They will reoccur unless they are iden- 
tified and eliminated. Special causes are signaled by data 
points that fall outside of the control limits, by consecutive 
points that fall above or below the process average, or by 
runs of increasing or decreasing points. 

Common causes of variation are inherent to the process; 
they are present at all times and effect the entire process. 
Common causes are reduced or eliminated through man- 
agement actions that change the process. 


3. STATISTICAL PROCESS CONTROL 
APPLICATION TO THE 
CONSTRUCTION OF SAMPLING FRAMES 
FOR PERIODIC SURVEYS 


3.1 United States Unemployment Insurance Benefits 
Quality Control 


The use of statistical process control as a quality man- 
agement tool for sampling frames is illustrated by an 
example from the United States Department of Labor’s 
Unemployment Insurance Benefits Quality Control program. 
Since 1987, the 50 states, the District of Columbia, and 
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Puerto Rico have conducted the Benefits Quality Control 
program in cooperation with the United States Depart- 
ment of Labor. The goal of the program is to reduce the 
overpayment and underpayment of Unemployment Insur- 
ance benefits by identifying the causes of payment errors 
and initiating measures to improve the benefit payment 
process. 

When an individual files a claim for Unemployment 
Insurance benefits, Unemployment Insurance staff deter- 
mine whether the claimant has met all of the eligibility 
requirements — for example, the claimant earned sufficient 
wages in his or her previous employment to qualify for 
benefits; the claimant is involuntarily unemployed; and the 
claimant is able and available to work and is actively 
seeking employment. If all of the eligibility requirements 
are satisfied, the state Unemployment Insurance agency 
issues a benefits check for the week of unemployment 
claimed. 


3.2 Benefits Quality Control Sampling Procedures 
and Sources of Error 


Each state selects weekly random samples of Unem- 
ployment Insurance payments that are examined to deter- 
mine if the correct amount was paid to the claimant. If the 
amount paid was incorrect, the investigator identifies the 
types and causes of the errors so that program managers 
can initiate corrective measures. The sampling frames 
are constructed each week from the universe of Unem- 
ployment Insurance payments that were issued between 
12:00 am Sunday and 11:59 pm the following Saturday. 
A computer program edits the state’s database to insure 
that only payments that meet the program’s operational 
definition of the target population are included in the 
frame. For example, payments for some temporary or 
small Unemployment Insurance programs are excluded 
from the frame. 

The volume of Unemployment Insurance checks issued 
each week (and therefore the size of the sampling frames) 
varies in response to the number of individuals who claim 
and receive benefits during that week. However, there are 
several sources of potential errors which can affect the 
integrity of the frame. Some of the most serious of these 
errors are: 


e The payments made from some of the local Unemploy- 
ment Insurance offices might not be picked up for 
inclusion in the state’s central database, due to tele- 
communication or ADP problems. 


e If the state builds a separate file for each day’s transac- 
tions, the transactions for one or more days might be 
erroneously omitted from the final cumulative file. 


e Incorrect coding of transactions could result in either 
foreign elements being included in the frame or the 
editing out of transactions that should be included. 
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4. DATA ANALYSIS AND MODEL 
DEVELOPMENT 


Figure | is a time series plot of sampling frame sizes for 
a 52 week period. Each week’s sampling frame consists of 
the previous week’s Unemployment Insurance benefit 
recipients who continue to receive benefits, minus the 
previous week’s Unemployment Insurance recipients who 
have returned to work, exhausted their benefits, or failed 
to file a claim, plus newly eligible claimants and eligible 
claimants who did not file a claim or were not compensated 
for a claim the previous week. 
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Figure 1. Number of UI payments per week. 


Control charts for individual observations assume that 
the data are independent and identically distributed (1.i.d.). 
However, if the data are serially correlated, the estimates 
of the process variance (and therefore the control limits) 
could be seriously in error. So, before control charts for 
the Unemployment Insurance sampling frame data can be 
constructed, we have to determine if the observations are 
serially correlated. 

The plot of the time series in Figure 1 provides visual 
evidence that the observations are not independent. The 
sampling frame data display distinct trends of increasing 
values during the first 13-week quarter, decreasing values 
over the next two quarters, and increasing values during 
the final 13-week quarter. The serial correlation suggested 
by the plot of the data in Figure 1 can be tested using methods 
developed to analyze time series. Although a detailed 
discussion of the analysis of time series data is beyond the 
scope of this article, the concepts of stationarity and auto- 
correlation will be examined, in order to explain the proce- 
dures used to identify the appropriate model. Readers who 
are unfamiliar with the basic principles of time series analysis 
should consult one of the many texts on the subject, in 
particular Box and Jenkins (1976). 
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4.1 Stationarity 


We can think of the individual observations that consti- 
tute a time series as a collection of jointly distributed 
random variables - p(z;, ..., Z,) — where p is a proba- 
bility density function and z,, ..., z, are random variables. 
If the joint distribution of the random variables does not 
Wary switirrespect. £0 timc, tat ds, P(Z;s ..1> can) = 
Die ne .s Stanam)> we process is said to be ‘sirictly 
stationary. In practice strict stationarity is difficult to 
establish. In this application, the time series is assumed to 
be weakly stationary. This is also referred to as second- 
order stationarity, because the first and second moments of 
the process are invariant with respect to time - E(z,) = 
E(Zi4m), VAR (Z) = VAR (Z4m), and COV (2,244) = 
COV (244m s214k+m)- 

Throughout the rest of this article, the terms stationary 
or stationarity refer to a process that satisfies the condi- 
tions of weak stationarity. 


4.2 Autocorrelation 


In a stationary time series the covariance between any 
two observations depends only on the number of time 
periods (lags) that separate them - COV(Z,,Z;4,) = 
COV (214 mZt+k4m)- The correlation of z, and z,, equals 
COV (2;52;4%)/VAR(z,) and is denoted p,, where k is the 
number of periods between observations. For example, p, 
is the correlation of observations in the time series sepa- 
rated by one period and equals COV (z,,Z,;4;)/WAR(Z,). 
A correlation for period k is referred to as an autocorrela- 
tion, because it is the correlation for observations which 
constitute a time series. The autocorrelations for the 
various lags can de displayed in a graph called a corre- 
logram, which is useful in identifying the appropriate 
model for a time series. 


4.3. Time Series Model Identification 


Figure 2 is the correlogram for the 52 week time series 
of the number of Unemployment Insurance payments in 
the sample frames. The autocorrelations decrease or ‘‘die 
out’’ very slowly, which is characteristic of a nonstationary 
process. (Again, the reader is referred to Box (1976) and 
other texts on time series for a complete discussion of 
model identification.) 

One method to transform a nonstationary series to a 
stationary series is differencing. The symbol B is the 
backshift operator, which when applied to z, shifts the 
subscript back one period. Thus, the first difference of z, 
ise(l = B) Zp 92 A: 

Figure 3 is the time series of the differences z, — z,_, of 
the Unemployment Insurance sampling frame data. This 
series appears stationary around a mean of zero. (The esti- 
mated sample mean of the differences is 150.8, with a stan- 
dard error of 2064.0. The test statistic = (150.8 — 0)/ 
2064 equals .07, and the hypothesis that 1» = O cannot be 
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rejected). First differences might not be sufficient to achieve 
stationarity for other time series, and transformations such 
as second differences - (1 — B)7z, =) (parece) = 
(Z;-1 — 2-2), seasonal differences, or logarithmic or 
other variance stabilizing procedures may be required. 
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Figure 2. Autocorrelations for UI weeks paid time series. 
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Figure 3. First differences of UI payments. 


The autocorrelations of the first differences of the time 
series, which are displayed in Figure 4, are consistent with 
a stationary process. The autocorrelations decrease rapidly, 
while the partial autocorrelations (not displayed) die off 
after lag 1. This suggests that the data can be modelled 
with a first-order integrated autoregressive process, ARI 
(1,1). The AR term indicates that a single autoregressive 
parameter will be estimated, and the integration term (1) 
shows that the original time series has been transformed 
using first differences. 
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Plot Symbols: Autocorrelations * Two Standard Error Limits . 
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Figure 4. Autocorrelations for first differences of UI weeks paid. 


4.4 Model Estimation 


The model was estimated using the ARIMA procedure 
of the SPSS Trends software (release 4.0), which is based 
on the work of Box and Jenkins. 


The tentative model is: 


Se ah nO) ia O47 eee, OL 
Gi 2(-. — Pia — fod) tees 


where @, is the first-order autoregressive parameter, and 
e is the error term, which is assumed to be normally 
distributed with a mean of 0 and variance o2. The 
estimated autoregressive parameter, ¢; is — .4045, and 
the estimated residual variance, or’ , 1S 184,275,853 (with 
50 degrees of freedom). The negative sign on the AR 
parameter is consistent with the alternating signs of the 
autocorrelations in Figure 4. The model does not include 
a constant term, because the estimated process mean was 


not significantly different than zero. 


4.5 Model Diagnostics 


The adequacy of the estimated model for the observed 
data can be assessed by examining the model residuals. If 
the model adequately fits the data, the residuals (e,) 
should be ‘‘white noise’’, that is, uncorrelated. Figure 5 
displays the autocorrelations of the model residuals. 
Although the autocorrelation at lag 13 in Figure 5 is signifi- 
cant, the Box-Ljung Q statistic through lag 13 is not 
significant. (The Q statistic tests the significance of 
autocorrelations for lags 1 through k. For a detailed 
discussion, see Box and Pierce (1970)). In addition, none 
of the partial autocorrelations (not displayed) are signifi- 
cant. These results indicate that the residuals are not 
serially correlated. 
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Plot Symbols: Autocorrelations * Two Standard Error Limits . 


Total cases: 52 Computable first lags: 50 


Figure 5. Autocorrelations for time series model residuals. 


To test the assumption that the model residuals are 
normally distributed, N (0:0): a Kolmogorov-Smirnov 
(K-S) goodness of fit test was conducted. For the esti- 
mated variance of 184,275,853, the K-S test statistic equals 
.591 (p = .876), and the hypothesis that the differences 
are normally distributed cannot be rejected. 

For a stationary AR (1) process, the absolute value of 
the autoregressive parameter must be less than one. To test 
the hypothesis that | ¢, | = 1 for the model, we compute: 
t = (| ¢ | — 1)/SE(¢{), where | ¢; | is the absolute 
value of the estimated autoregressive parameter, and 
SE(¢ , ) is the standard error of ¢;. The model statistics 
resultint = (.4045 — 1)/.1295Sort = —4.6. The chance 
of observing an absolute value of ¢; as small as .4045 if 
the true absolute value of ¢,; = 1 is very small (< .00001). 
The hypothesis that | ¢, | = 1 is rejected, and we can 
conclude that the series of first differences is stationary. 


5. USE OF THE ARIMA MODEL IN 
A CONTROL CHART 


5.1 Control Charts for Individual Observations 


The control limits for a chart of individual observations 
are set at ¥ + 30’, where X is the average of observation 
values and o’ is the estimated standard deviation of the 
process. Ryan (1989) discusses alternative procedures to 
estimate the process standard deviation either by computing 
the average of the moving ranges (the mean of the absolute 
differences of successive observations) or using the stan- 
dard deviation (s) of the sample observations, 0’ = s/c, 
where c is an adjustment constant which depends on the 
sample size. 

When data are serially correlated, the use of either the 
sample standard deviation or the average moving range 
can result in poor estimates of o. The control limits 
constructed from these estimates can produce seriously 
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misleading results by either generating false signals that 
the process is out of control or failing to detect special 
causes of process variation. The moving range can under- 
estimate o, because the differences of successive values will 
tend to be small if the successive observations are highly 
correlated. The underestimation of o will result in control 
limits that are too narrow and an increase in the number 
of signals of special causes. Ryan notes that using the 
sample standard deviation to estimate the process standard 
deviation will result in a better estimate of o than the aver- 
age moving range when the data are correlated, provided 
the sample consists of at least 50 observations. However, 
the sample standard deviation is an unbiased estimator of 
o only when the observations are independent. 

Vasilopoulos and Stamboulis (1978) analyzed the effect 
of serially correlated data on the control limits of < and 
s (standard deviation) charts and developed equations for 
factors that can be used to adjust the control limits for data 
generated by an autoregressive process. Alternatively, a 
time series model can be identified for the correlated data, 
and a control chart can be constructed using the model 
residuals to monitor the process. This approach is described 
by Berthouex, Hunter, and Pallesen (1978) for subgroups 
of measurements of environmental data collected at water 
treatment plants. Alwan and Roberts (1988) use the resid- 
uals of exponentially weighted moving average (EWMA) 
models for both stationary and nonstationary time series. 
Montgomery and Mastrangelo (1991) use the residuals of 
an autoregressive model in an EWMA chart and contend 
that EWMA charts can be used to approximate many 
autocorrelated models, particularly if the observations are 
positively correlated and the mean does not drift too 
quickly. The reader is also referred to Maragah and 
Woodall (1992) and Woodall and Faltin (1993) for addi- 
tional discussion of the effects of autocorrelation on 
statistical process control procedures. 


5.2 Control Charts for the Unemployment Insurance 
Data 


Figure 6 is a control chart of the residuals (e, = z, — z/) 
of the ARI (1,1) model identified for the Unemployment 
Insurance sampling frame data. Since the model diagnostics 
support the conclusion that the residuals are independent 
and identically distributed (i.i.d.) N(0,02), the residuals 
are standardized, so that the chart’s center line is 0 and 
the control limits are set at + 3. The chart includes model 
residuals for the sampling frame sizes in the 52 week 
baseline period and subsequent calendar quarter. The 
difference between the size of the sampling frame for week 
56 and the value predicted by the model falls outside the 
upper control limit, signaling a special cause. 

As an alternative to charting the model residuals, control 
charts for the Unemployment Insurance sampling frame 
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Figure 6. Control chart for model residuals (baseline data + 
next quarter). 


sizes can be constructed. The original observations must 
be transformed to achieve stationarity, if necessary. The 
estimated parameters of the time series model are used to 
construct the mean and control limits of the chart. The 
variance of an AR(1) process is 0? = 02/(1 — $7). For 
the time series model of first differences, ¢; is — .4045, 
and the estimated residual variance, o; 2 is 184,275,853. 
The estimated process variance is 184,275,853/(1 — .1636) 
or 220,325,579.4, and the process standard deviation is 
14,843.4. The upper and lower control limits are set at 
+30’ from the estimated mean difference of zero: 
+ 44,530.2. The control chart is shown in Figure 7 and 
signals a special cause for observation 56, like the control 
chart for the residuals in Figure 6. 
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Figure 7. Control chart for UI payments (first differences — 
baseline + next quarter). 
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6. CONCLUSIONS 


Statistical process control is a useful quality assurance 
tool for surveys in which samples are selected from frames 
that are constructed for specified periods from a contin- 
uous process. Because the frame sizes constitute a time 
series, the data may be serially correlated and may have 
to be transformed in order to achieve stationarity. If the 
observations are correlated, the appropriate time series 
(ARIMA) model must be identified in order to estimate 
the process variance used in setting the control limits. The 
time series in the preceding example was fitted by a first- 
order autoregressive integrated (differenced) model - ARI 
(1,1). More generally, time series may be described by 
other ARIMA (p,d,q) models, where p is the number of 
autoregressive terms in the model, d is the degree of 
differencing to achieve stationarity, and g is the number 
of moving average terms in the model. Seasonal time series 
models include additional AR, MA, and differencing 
parameters for the appropriate lag(s). 

Once the model has been identified from baseline data, 
observations from subsequent periods can be plotted in 
the control chart. In the control charts in Figures 6 and 7, 
one calendar quarter (13 weeks) of observations are plotted 
following the observations from the 52 week baseline. The 
time series model should be checked periodically, depending 
on the data collection interval, to determine if the model 
parameters have changed. 

If the statistical process control procedures signal a 
special cause of variation, survey managers must use other 
quality management tools to determine the root causes of 
the frame problems and then implement corrective actions 
to improve survey procedures. Survey managers can move 
from troubleshooting and error correction to continuous 
improvement of the survey process by systematically 
removing the assignable causes of variation identified 
through statistical process control. 

In the case of the Unemployment Insurance sampling 
frame data, the special cause was not preventable: the 
volume of Unemployment Insurance payments spiked 
during a week which followed a short work week due to 
a holiday and which coincided with a layoff at a large 
establishment. The large sampling frame was not the result 
of a technical problem with the construction of the frame. 
In other states, at different time periods, statistical process 
control has detected errors as diverse as data entry mistakes 
(a frame of 558,432 reported instead of 5,558,432), 
omission of the Unemployment Insurance transactions for 
one of five work days, resulting in an approximate 
20 percent decrease in the frame size, and the failure to 
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update edits in the sample selection software, which caused 
“oreign elements to enter the frame. 

The procedure described in this article is applicable to 
other areas of survey and information management in 
addition to the integrity of sampling frames. The proce- 
dure can be used to reduce nonsampling error attributable 
to data recording or data entry for surveys conducted 
daily, monthly, efc. More generally, statistical process 
control can be used to assure the integrity of databases or 
management information systems whenever information 
is collected or reported in subgroups, such as data collected 
at multiple sites or by several researchers or auditors. 
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